Skip build-failure-analysis agent on green builds

[Evangelink]

Summary

The Build Failure Analysis workflow keeps producing red ❌ checks on PRs with perfectly green builds. Example: https://github.com/microsoft/testfx/actions/runs/26760910013.

PR #8726 (merged earlier today) only suppressed the auto-tracked tracking issue; the failed check on the PR itself was unaffected.

Root cause (confirmed from run logs)

1. The workflow uses gh-aw — there is no agent-job-level if: hook, so the agent runs on every PR.
2. On a green build, the agent's instructions say to call the noop safe-output and stop.
3. The Copilot CLI hits a transient mid-conversation flake (Response was interrupted due to a server error).
4. gh-aw's copilot_harness.cjs hard-codes MAX_RETRIES = 3 and retries with --continue — each retry re-issues the terminating noop call.
5. Cumulative count: 4 noops. The default cap is 1, so noops 2–4 are rejected, the agent step exits 1, and the PR shows a red X.

Fix

Split the workflow so the actual build runs in a dedicated build job whose outputs gate the AI agent pipeline:

• New jobs.build: runs ./build.sh --binaryLog, locates the binlog, installs Microsoft.AITools.BinlogMcp, dumps JSON to /tmp/binlog-data/, and uploads build-failure-analysis-data as an artifact (only on failure). Outputs: outcome, binlog-found, binlog-relative-path. Build step uses continue-on-error so the job itself always succeeds, and exit + "" + \ + "" + prevents a tee glitch from misclassifying a green build as failed.
• on.needs: [build] makes pre_activation / activation wait for build.
• Top-level if: needs.build.outputs.outcome == 'failure' gates activation — and the agent cascades to skipped — so the AI pipeline never runs on a green build.
• The agent job downloads the artifact instead of rebuilding, and only installs NuGet.Mcp.Server (which it uses at runtime for NU#### errors).

For the residual case where the agent still runs (failed builds), I raised the safe-output caps to absorb the harness retry budget:

• noop.max: 5 (1 happy-path + 4 retry-amplified)
• add-comment.max: 5 (hide-older-comments: true collapses duplicates)
• create-pull-request-review-comment.max: 25 (shared body asks for top-5 issues, so 5 × 4 retries = 20)

Slash-command variant

Applied the same split to build-failure-analysis-command.md, plus an explicit refs/pull/<n>/merge checkout in its build job — without it, pull_request_comment events check out the default branch instead of the PR a maintainer ran /analyze-build-failure on.

Verification

• Compiled both workflows with gh aw compile --strict (gh-aw v0.75.4): 0 errors, 0 warnings.
• Inspected the generated .lock.yml files to confirm:
  • pre_activation, activation, and agent all gate on needs.build.outputs.outcome == 'failure'.
  • Build job emits the expected outputs and uploads the artifact only on failure.
  • Agent job downloads the artifact and exports GH_AW_BUILD_OUTCOME / GH_AW_BINLOG_PATH correctly.
  • conclusion: job's if: is not affected (only runs when agent did not skip or activation lockdown failed).

Risk

Low. The change is structural — the agent prompt body (shared/build-failure-analysis-shared.md) is unchanged. The build job runs ./build.sh --binaryLog identically to before (same script, same flags) and the AI agent receives identical inputs (same JSON dumps, same env vars). Worst case the new gating is wrong and the agent doesn't run on a failed build — but that just makes the workflow silently no-op, never a regression vs the current red-X state.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR restructures the gh-aw “Build Failure Analysis” workflows so the Copilot agent pipeline is skipped on successful (green) builds, preventing transient AI flakes from surfacing as red ❌ checks on otherwise healthy PRs.

Changes:

• Split both workflows into a dedicated build job (produces binlog + JSON dump and uploads an artifact on failure) and an agent pipeline gated on needs.build.outputs.outcome == 'failure'.
• Updated the agent job to download the build artifact instead of rebuilding, and adjusted exported GH_AW_* context accordingly.
• Increased safe-output caps to tolerate Copilot CLI harness retries without safe-output validation failures.

Show a summary per file

File	Description
.github/workflows/build-failure-analysis.md	Adds a separate build job and gates activation/agent execution on build failure; switches agent setup to artifact download.
.github/workflows/build-failure-analysis.lock.yml	Regenerated lock workflow reflecting new job graph, gating, and safe-output limits.
.github/workflows/build-failure-analysis-command.md	Mirrors the same build/agent split for the slash-command workflow, including explicit PR merge-ref checkout.
.github/workflows/build-failure-analysis-command.lock.yml	Regenerated lock workflow reflecting the command variant’s new job graph and gating.
.github/aw/actions-lock.json	Updates action pin entries used by gh-aw compilation (notably actions/setup-dotnet@v5).

Copilot's findings

• Files reviewed: 5/5 changed files
• Comments generated: 4