[agentrx-optimizer] Daily Workflow Optimization - 2026-07-22

[github-actions]

Executive Summary

AgentRx normalized 29 distinct failing agentic-workflow runs (from report-failure-as-issue records across ~26 workflows in github/gh-aw) into trajectory IR and ran the deterministic invariant checker. Grouping the normalized failure telemetry by root cause, the single largest actionable cluster is terminal API errors handled by a retry storm: model-API 403 ai_credits_limit_exceeded (AI‐credit budget exhaustion) and 429 rate‐limit errors that end in engine termination. The top finding: on a terminal 403 ai_credits_limit_exceeded, the engine re‐issues the same request up to 11 times before failing — pure wasted wall‐clock/retry budget that also worsens rate‐limit pressure. gh‐aw's harness already classifies this error as non‐retryable, yet the 11‐attempt storm still occurs, proving the retries happen inside the CLI/SDK invocation, below the harness guard's reach.

AgentRx Evidence

• Critical step: the model-API request step (IR step where content is tagged [CRITICAL FAILURE STEP]) returning 403 ai_credits_limit_exceeded / 429, followed by an in‐CLI retry loop.
• Failure category (root cause): resource_budget_exhaustion / model_rate_limiting → engine_terminated_on_api_error. Fix type: improving retry/backoff strategy (fail‐fast on terminal errors).
• Frequency / impact: retry/backoff‐addressable runs = ai_credits_exhausted (4) + rate_limit (1) + permission (1) + engine_failure primary (3) ≈ 9 of 29 failing runs (~31%); engine_failure appears on 7 runs when counted multi‐label. This is the largest actionable fix‐type group. (For comparison: missing_tools 5, timeout 5, unclassified generic_failure 10.)
• Retry‐storm evidence: run 29896892294 — used 1030.9 / 1000 AI credits, 403 ai_credits_limit_exceeded, attempt 1/11; run 29826783727 — attempt 1/11; run 29874295528 — 429 → engine failure, 5 attempts.
• Representative runs: 29896892294, 29826783727 (Daily VulnHunter Scan); 29874295528 (Daily BYOK Ollama Test).

AgentRx Artifacts

IR summary — 29 trajectories loaded, 29 valid, 29 steps; domain auto‐detected flash; deterministic normalization (no LLM fallback needed). Each trajectory maps one failing run's session record to an ordered start → execute → [CRITICAL FAILURE STEP] sequence.

Invariant / checker highlights — deterministic checker ran across all 29 trajectories and reported 0 auto‐generated invariant violations. This is a limitation, not a clean bill of health: static/dynamic invariant generation and the LLM judge require an LLM endpoint (copilot/azure/trapi), and none was reachable in this run (copilot CLI unauthenticated + egress firewall). The static stage failed with Static invariants JSON parse failed (empty LLM response); report was skipped (No judge output to report on). Completed stages: ir, check.

Judge classification (deterministic, evidence‐derived — LLM judge unavailable) — classification was derived from the normalized IR run‐session telemetry rather than the LLM judge:

violation	evidence	fix_type	rationale
ai_credits_exhausted (4)	run 29896892294 used 1030.9/1000 credits, 403 ai_credits_limit_exceeded, retried up to 11×	improving retry/backoff strategy	Terminal 403 treated as retryable → 11‐attempt storm; short‐circuit non‐recoverable quota errors
rate_limit (1)	run 29874295528 — HTTP 429 → engine failure	improving retry/backoff strategy	429 needs exponential backoff + jitter, not immediate re‐fire
engine_failure (3 primary / 7 multi‐label)	engine terminated after API errors (predominantly 403/429)	improving retry/backoff strategy	Correct retry classifier addresses the shared upstream cause
permission (1)	run 29843908934 — permission denied	improving retry/backoff strategy	Permission‐denied is terminal; must not be retried
missing_tools (5)	run 29896126198 — "MCP toolsets unavailable in runner - tools not loaded"	precondition checks before expensive tools	Pre‐flight MCP‐reachability check fails early with a clear message
timeout (5)	run 29848004889 — cancelled after exceeding time limit	reducing token‐heavy context payloads	Trim context / split work to finish within limit
generic_failure (10)	plain "Workflow Failure", no guardrail signal	add missing telemetry attributes for triage	Undiagnosable from run‐session records alone

Known limitations — (a) LLM‐gated stages (static/dynamic/judge/report) did not run; classification is evidence‐derived. (b) Per‐turn token/turn/duration telemetry was unavailable: the agentic-workflows logs/audit MCP tools could not download agent artifacts within the gateway's hard ~60s per‐tool deadline, so trajectories were built from report-failure-as-issue run‐session records (run id, workflow, engine, error class, credit numbers, retry counts, missing‐tool notes) — the runs[] session fields the mission prioritizes. (c) One of the 4 credit‐exhaustion runs is Daily Max Ai Credits Test, a workflow that intentionally exercises the limit; the other 3 (incl. two real Daily VulnHunter Scan runs) are genuine.

Recommended Optimization

One change — fail fast on terminal API errors instead of retrying them. Cap/disable the engine's inner HTTP‐retry budget for non‐recoverable error classes — 403 ai_credits_limit_exceeded (and the sibling quota/budget 403s) and permission‐denied — so the CLI exits immediately and hands off to the existing harness guard, rather than burning up to 11 in‐process attempts. Pair it with proper exponential backoff + jitter for genuinely transient 429.

• Why highest impact: it is the largest actionable, evidence‐backed cluster (~9/29 runs, 7 multi‐label engine_failure), and the smallest meaningful change: gh‐aw already detects the credit error at the harness layer (AI_CREDITS_EXCEEDED_PATTERNS in actions/setup/js/harness_retry_guard.cjs — verified to match the real error string), so the only gap is that the retry storm occurs inside the CLI invocation before the harness sees output. Eliminating those inner retries wastes no model credits (they return 0 tokens) but saves runner‐minutes and stops the storm from amplifying 429 pressure for the rate‐limit cluster.
• Where to implement:
  • actions/setup/js/harness_retry_guard.cjs — extend the terminal‐error guard so the non‐retryable classification is enforced against the inner retry path (and cannot be silently bypassed); add a regression case to actions/setup/js/harness_retry_guard.test.cjs.
  • pkg/workflow/claude_engine.go + pkg/workflow/engine_helpers.go (applyEngineHarnessRetryEnv, GH_AW_HARNESS_MAX_RETRIES / GH_AW_HARNESS_BACKOFF_MULTIPLIER) — drive the inner engine/SDK retry budget to a low value for terminal‐class errors.
  • Optionally the api‐proxy that emits the 403 ai_credits_limit_exceeded — return it so the CLI/SDK treats it as non‐retryable.

Validation Plan

• How to confirm: on the next runs of Daily VulnHunter Scan / Daily Max Ai Credits Test, the failure log should show a single API attempt on 403 ai_credits_limit_exceeded (no attempt 1/11), with the structured "AI Credits Budget Exceeded" safe‐output still emitted.
• Expected metric changes: retry‐storm runs (retry_attempts_max ≥ 5) drop toward 1; mean time‐to‐failure for the credit/permission cluster falls sharply (fewer wasted attempts / runner‐minutes); 429‐driven engine_failure runs recover more often once backoff+jitter is applied. Re‐running this AgentRx pipeline should show the improving retry/backoff strategy cluster shrink relative to the 29‐run baseline.

References

• §29896892294 — Daily VulnHunter Scan, 1030.9/1000 credits, 403, 11 attempts
• §29826783727 — Daily VulnHunter Scan, 11 attempts
• §29874295528 — Daily BYOK Ollama Test, 429 → engine failure

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by ⚡ Daily AgentRx Trace Optimizer · age00 959.7 AIC · ⌖ 36.3 AIC · ⊞ 6.4K · ◷

• expires on Jul 28, 2026, 10:58 PM UTC-08:00