[agentrx-optimizer] Daily Workflow Optimization - 2026-07-20

[github-actions]

Executive Summary

AgentRx analyzed 45 recent gh-aw agent session runs (downloaded via the agenticworkflows MCP logs tool). Of these, 10 runs failed (22% failure rate). The single dominant, cross-cutting finding: failing runs are effectively un-triageable because they emit no agent-step telemetry — 10 of 10 failing runs have an empty token_usage.jsonl, and turn-level attributes (Turns, AvgTimeBetweenTurns) are unpopulated across all 45 runs (0/45). The critical trajectory step (agent execution, Step-2) is a black box on every failure, so no failure — including the workflow that failed 3× — can be root-caused from the default telemetry.

The highest-impact optimization is adding/guaranteeing agent-step telemetry capture on failure so the failing 22% become diagnosable.

AgentRx Evidence

• Critical step: Step-2 Agent execution (per the AgentRx IR trajectory model; index 2 of 4). For failing runs this step carries zero measured tokens, turns, or evals.
• Failure category: Missing/insufficient observability → un-attributable failure (telemetry gap). Secondary: token-heavy context payload; task-success-vs-run-failure mismatch.
• Frequency / impact: 10/45 runs failed (22%). 10/10 failing runs have empty token_usage.jsonl; 9/10 have no agent telemetry at all. Smoke CI failed 3× (highest single-workflow failure count) yet cannot be diagnosed. Turns populated in 0/45 runs.
• Root cause (code-grounded): The usage artifact (token_usage.jsonl, evals.jsonl) is captured by the conclusion job — see pkg/cli/logs_run_processor.go:179-184 ("Failure-conclusion runs are kept (with empty metrics)... No artifacts available for run") and :250 ("included in the usage artifact by the conclusion job"). When a run fails before/at the conclusion job, that artifact is never produced, so failing runs land with empty metrics by construction.
• Representative run IDs: 29720659363 (Smoke Copilot), 28628300645 / 28622273766 / 28623371404 (Smoke CI ×3), 29720566806 (Tidy).

AgentRx Artifacts

Pipeline run: runs/gh-aw-daily/ — domain auto-detected flash, endpoint copilot.

IR summary (Stage 1/6 — completed):

• 45 trajectories loaded, 45/45 valid, non-degenerate (no LLM fallback; ir_used_llm_fallback=false).
• Each run normalized to a 4-step trajectory: Step-1 dispatch → Step-2 agent execution → Step-3 firewall/network → Step-4 outcome/evals.
• Output: runs/gh-aw-daily/trajectory_ir.json.

Invariant / checker / judge stages (2–6): not available in this environment. static failed with Static invariants JSON parse failed because no LLM endpoint is reachable from the sandbox (the generator received a non-JSON error response); dynamic, check, and judge depend on static, and report skipped (No judge output to report on). Per the fallback procedure, the recommendation below is grounded in the completed IR plus deterministic session-telemetry analysis rather than LLM-classified invariant violations.

Deterministic telemetry analysis (substitute for check.json/report):

• 45 runs; 10 failures (22%); failures per workflow: Smoke CI:3, then Smoke Copilot, Tidy, Deployment Incident Monitor, Copilot PR Prompt Pattern Analysis, PR Sous Chef, Daily Project Performance Summary Generator, Daily BYOK Ollama Test (1 each).
• 10/10 failing runs: empty token_usage.jsonl; 0/45 runs: Turns>0.
• Firewall: 17 runs had blocked requests; pi.dev:443 blocked in 4 runs (pi-engine egress denied).
• Heaviest run: Smoke Copilot — 2,608,968 input tokens (2,474,496 cache-read) in a single request, yet all evals passed (smoke-passed=YES, build-succeeded=YES, issue-created=YES) while the run conclusion is failure.

Known limitations: LLM-dependent invariant generation/judging unavailable (no reachable endpoint); default logs artifact set is usage only, so agent-step stdout for failing runs was not retrievable from cache.

Labeled violations (fix-type taxonomy):

violation	evidence	fix_type	rationale
Failing runs emit no agent-step telemetry	10/10 failing runs have empty token_usage.jsonl; 9/10 zero usage	adding missing telemetry attributes for better triage	usage artifact gated on conclusion job (logs_run_processor.go:250); failures skip it → un-triageable
Turn-cadence attributes never populated	Turns/AvgTimeBetweenTurns = 0 in 45/45 runs	adding missing telemetry attributes for better triage	no step-level cadence → cannot localize the critical/stalling step
Oversized single-request context	Smoke Copilot: 2.59M input / 2.47M cache-read tokens in 1 request	reducing token-heavy context payloads	inflates cost (~118 AIC/run) and latency; near context ceiling
Task-success vs run-failure mismatch	Smoke Copilot evals YES/YES/YES but conclusion=failure	adding precondition checks before expensive tools	post-agent step failure is not attributed to a step
pi-engine endpoint blocked	pi.dev:443 blocked in 4 runs	adding precondition checks before expensive tools	egress denied mid-run wastes the run; validate allowlist before dispatch

Recommended Optimization

Guarantee agent-step telemetry is emitted and uploaded on failure, decoupled from the conclusion job.

• One specific change: Emit/upload the agent usage telemetry (token_usage.jsonl + the turn-count / AvgTimeBetweenTurns attributes) from an if: always() step in the agent job itself, rather than relying on the conclusion job (which does not run when the agent job fails). At minimum, always write a stub token_usage.jsonl and populate Turns/AvgTimeBetweenTurns for every run so a failed run is distinguishable from a run that never started.
• Why highest impact: It is the prerequisite for diagnosing the entire 22% failure class. Today the most-frequent failure (Smoke CI, 3×) cannot be root-caused at all. Every other candidate fix (retry tuning, context trimming) is un-verifiable while failing runs remain black boxes. This is the smallest change that unblocks all downstream triage.
• Where to implement: The usage-artifact capture path is documented in pkg/cli/logs_run_processor.go:179-260 (conclusion-job dependency) and metric extraction in pkg/cli/logs_metrics.go (Turns, TokenUsage). The generated upload step lives in the compiler's artifact emission (pkg/workflow/publish_artifacts.go / engine log steps, which already use if: always() for other captures) — extend the same always-on upload to the agent usage telemetry.

Validation Plan

• How to confirm: On the next scheduled cohort, re-run this analysis. For failed runs, token_usage.jsonl should be non-empty and Turns/AvgTimeBetweenTurns should be populated (> 0 or an explicit 0 with a captured failure reason).
• Expected metric changes: failing runs with empty token_usage.jsonl drops from 10/10 → ~0/10; runs with Turns populated rises from 0/45 → ≈100%. This should let a subsequent AgentRx judge pass (once an endpoint is available) attribute each Smoke CI failure to a concrete step, turning the 22% failure rate into an actionable, per-step breakdown.

References

• §29720659363 — Smoke Copilot (evals passed, run failure, 2.59M-token single request)
• §28628300645 — Smoke CI (one of 3 un-triageable failures)
• §29720566806 — Tidy (failure, evals UNKNOWN, empty telemetry)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by ⚡ Daily AgentRx Trace Optimizer · 317.2 AIC · ⌖ 35.2 AIC · ⊞ 6.2K · ◷

• expires on Jul 26, 2026, 10:49 PM UTC-08:00

[pelikhan]

/q update the logs logic to specify to download the "agent" artifact

[github-actions]

🎩 Mission equipment ready! Q has optimized your workflow. Use wisely, 007! 🔫

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #47217

View newer issue

---

This action was performed automatically by the Daily AgentRx Trace Optimizer workflow.