Add flaky-test detection and auto-fix agentic workflows

[ViktorHofer]

Summary

Adds a flaky-test detection and auto-fix system for msbuild's public CI, built on GitHub Agentic Workflows (gh-aw). The design ports the approach used by dotnet/fsharp, adapted to msbuild's anonymous-accessible dnceng-public pipeline.

Three pieces:

1. Detector skill — .github/skills/flaky-test-detector/

Get-FlakyTests.ps1 scans recent failed builds on the public msbuild-pr pipeline (definition 75) using only anonymous Azure DevOps APIs (build list, timeline, artifact zip). It uses an evidence-source model: a test is flagged flaky when it fails across >= MinSources (default 3) distinct sources, where a source is either:

• an approved, non-draft PR whose base is main (all its failed validation builds collapse to one source), or
• a single rolling/CI build on main (each is independent evidence, since main should be green).

It parses TRX results, aggregates by normalized test name, and emits an evidence-grade JSON report with a scanComplete flag. Smoke-tested against live CI.

2. Daily triage-and-fix workflow — .github/workflows/flaky-test-detector.agent.md

A single workflow runs daily (+ manual dispatch) and does the whole job end to end:

1. Runs the detector and applies guard rails (a truncated/biased scan does nothing).
2. Classifies each flagged test as a likely flake vs. a likely regression (see safety properties) and never quarantines a regression.
3. Files/updates one flaky-test tracking issue per likely-flake, deduped via a hidden <!-- flaky-test-id: --> marker.
4. Selects the day's candidates, builds the repo once, and screens all of them for reproducibility in parallel — each candidate's prebuilt test assembly is run directly (not via dotnet test), which avoids the non-Windows --coverage injection so the loops don't contend. Reproduced tests then get a minimal determinism fix (authored sequentially and time-boxed, for up to the 5 strongest, test-only where possible); everything not fixed gets a quarantine with [ActiveIssue("<issue url>")] (from Microsoft.DotNet.XUnitV3Extensions) — never [Fact(Skip=...)].
5. Accumulates all of the day's edits and opens exactly one combined draft PR containing every quarantine and determinism fix for that run, then comments the outcome on each tracking issue.

Quarantine backlog (def 344 as a data source): the same daily workflow also consumes the quarantine re-validation pipeline (AzDO definition 344, see §3) to work down the existing backlog. It re-runs the same detector against def 344 with a new -IncludePassed mode (no separate script — the quarantine pipeline mirrors the build pipeline, so the detector's failing-set aggregation already yields the "still-flaky in 344" signal; -IncludePassed adds the passing-test signal). From that it (a) un-quarantines tests that have gone consistently green — removing [ActiveIssue] only when scanComplete, the test is green across >= 4 distinct quarantine builds spanning >= 3 days on every platform it's quarantined for, and it never failed in the window — and (b) re-attempts determinism fixes on tests still flaking there (when reproducible on the runner). Both fold into the same combined draft PR under independent caps (<= 5 un-quarantines, <= 3 backlog fixes per run). Un-quarantining relies on def 344's real-CI evidence, not a local pass.

Caps per run: <= 5 new issues, <= 12 comments, exactly one combined draft PR. The PR patch is restricted to src/**/*.cs (allowed-files) and never touches .github/** (excluded-files + a pre-PR git diff check). Fixes # is used only for real determinism fixes (auto-closes the issue on merge); quarantines use Tracked by # so the tracking issue stays open.

Cross-run dedup: if a previous run's combined PR is still open and unmerged, the next run fetches the open flaky-test PR bodies and does an exact local marker match, so already-in-flight tests are skipped rather than re-quarantined (their quarantine lives on the open PR's branch, not yet on main).

3. Quarantine re-validation pipeline — azure-pipelines/quarantine.yml

[ActiveIssue] stamps the Category=failing trait, which normal CI excludes — so a quarantined test stops producing pass/fail signal. This scheduled AzDO pipeline runs only the quarantined tests (RunQuarantinedTests=true flips the filter to --filter-trait Category=failing) twice daily on main, across Windows, Linux and macOS (mirroring the per-OS test legs of .vsts-dotnet-ci.yml), so quarantines — including platform-scoped ones — keep producing data. Quarantined-test failures don't fail the pipeline (--ignore-exit-code 8;2); the per-test signal lives in the published test results, so a red run means a real infra problem. A test that goes consistently green there is a candidate to un-quarantine — and the daily workflow now acts on that signal automatically (see §2, "Quarantine backlog"). These quarantine-only runs also skip code coverage — Directory.Build.targets gates the non-Windows --coverage args off RunQuarantinedTests, since coverage instrumentation adds overhead and can perturb the timing-sensitive flakes being measured.

Safety properties

• Only the main branch is considered (detector, the agentic workflow, and the quarantine pipeline schedule).
• Regression vs. flake guard: a single stable error signature dominated by rolling-main failures (or PR failures that only began after main went red), or an N/N deterministic local repro, is treated as a likely regression — not quarantined or auto-fixed, flagged for human triage instead. This avoids masking real bugs.
• Determinism fixes prefer test-only changes; non-trivial product-code changes are deferred to humans (the test is quarantined instead). Quarantine is the safe default when reproduction isn't achievable.
• The accumulated patch is validated before the PR opens (allowed-path git diff check + compiling the changed test projects).
• Tight issue/comment/PR caps; one combined draft PR per run for human review.
• Uses the same mandatory PAT-rotation frontmatter convention as the existing close-stale-prs workflow.

Notes

• Lock file compiled with gh-aw v0.68.1 (schema v3), matching the repo's existing *.agent.lock.yml.
• This is opt-in automation that only creates issues and draft PRs for human review — it does not change any product or build behavior. (The quarantine pipeline runs existing tests on a schedule.)
• Rollout: committing quarantine.yml does not by itself create a scheduled run — an Azure DevOps pipeline definition pointing at it must be created/enabled once.

Security review (new secrets/actions introduced)

The workflow uses the established PAT-pool secrets (COPILOT_GITHUB_TOKEN, COPILOT_GITHUB_TOKEN_1..9) and the github/gh-aw-actions/setup action — identical to the already-approved close-stale-prs.agent.md workflow. No new external actions or secrets beyond that approved set.

[github-actions]

🔍 Skill Validator Results

⚠️ Warnings or advisories found

Scope	Checked
Skills	1
Agents	0
Total	1
Severity	Count
---	---:
❌ Errors	0
⚠️ Warnings	1
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	Found 1 skill(s)
ℹ️	[flaky-test-detector] 📊 flaky-test-detector: 3,457 BPE tokens [chars/4: 3,346] (standard ~), 13 sections, 5 code blocks
ℹ️	[flaky-test-detector] ⚠ Skill is 3,457 BPE tokens (chars/4 estimate: 3,346) — approaching "comprehensive" range where gains diminish.
ℹ️	✅ All checks passed (1 skill(s))

Full validator output

```text Found 1 skill(s) [flaky-test-detector] 📊 flaky-test-detector: 3,457 BPE tokens [chars/4: 3,346] (standard ~), 13 sections, 5 code blocks [flaky-test-detector] ⚠ Skill is 3,457 BPE tokens (chars/4 estimate: 3,346) — approaching "comprehensive" range where gains diminish. ✅ All checks passed (1 skill(s)) ```

[Copilot]

Pull request overview

This PR introduces an opt-in, agentic automation system (via GitHub Agentic Workflows / gh-aw) to detect flaky tests from public Azure DevOps CI signals, file/update flaky-test tracking issues, and (for strong candidates) dispatch a fixer workflow that opens a single draft PR containing either a minimal determinism fix or a quarantine (Skip).

Changes:

• Add a scheduled “Flaky Test Detector” workflow that runs daily, produces/consumes a structured flaky-test JSON report, and files/updates issues + optionally dispatches a fixer.
• Add a dispatched “Flaky Test Fix” workflow that re-verifies flakiness, attempts local reproduction, and opens at most one draft PR.
• Add a flaky-test-detector skill and PowerShell implementation that scans anonymous AzDO APIs + TRX artifacts to aggregate failures by “distinct evidence sources”.

Show a summary per file

File	Description
.github/workflows/flaky-test-fix.agent.md	gh-aw workflow definition + agent prompt for dispatched fixer.
.github/workflows/flaky-test-fix.agent.lock.yml	Compiled gh-aw lock file for fixer workflow.
.github/workflows/flaky-test-detector.agent.md	gh-aw workflow definition + agent prompt for scheduled detector/triage.
.github/workflows/flaky-test-detector.agent.lock.yml	Compiled gh-aw lock file for detector workflow.
.github/skills/flaky-test-detector/SKILL.md	Skill documentation describing the evidence-source model + conventions.
.github/skills/flaky-test-detector/scripts/Get-FlakyTests.ps1	Detector implementation: fetch failed builds, download TRX artifacts, aggregate evidence, emit JSON report.

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 5

[github-actions]

Expert Code Review — Flaky Test Detector

Overall assessment: Well-designed PR — solid security posture, clear documentation, good separation of concerns.

Strengths

• Security hardening is excellent: DTD processing prohibited, XmlResolver = null on XML parsing, zip PK-header validation, size caps on artifacts/TRX files, $MaxTrxPerArtifact limit, artifact name sanitization for paths, and proper [uri]::EscapeDataString for URL construction.
• The evidence-source model (collapsing PR builds, treating rolling builds independently) is well-reasoned and clearly documented.
• Temp directory cleanup in a finally block ensures no orphaned files.
• The scanComplete guard (propagated to callers via JSON) is a good design for preventing action on incomplete data.
• Workflow agent definitions follow the existing repo patterns (PAT pool selection, network allowlist, etc.) and are consistent with review.agent.md.
• SKILL.md is thorough and well-structured — excellent reference for future maintainers.

Issues Found (see inline)

1. ReadAllBytes for 2-byte header check — allocates potentially 200MB per artifact just to read 2 bytes. Use a stream.
2. $scanComplete false-negative — relies on $buildsResponse.count which may be absent from AzDo API responses, silently marking truncated scans as complete.
3. Substring(0, 10) on external data — minor robustness concern on the Get-BuildDate helper.

Not Flagged (positive notes)

• No MSBuild engine code is modified — no backwards compat, ChangeWave, or API surface concerns.
• Lock files are auto-generated and consistent with the .md definitions.
• Permissions are appropriately scoped (read-only for the detector, broader for the fixer which needs to create PRs).
• The excluded-files: [".github/**"] in the fixer workflow's create-pull-request config is a good defense-in-depth measure.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #13915 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #13915 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Expert Code Review (on open) for issue #13915 · ● 17.8M

[Copilot]

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 1