Fix flaky detector: unblock dnceng feed, fail fast, and defer quarantine to a second run

[ViktorHofer]

Problem

Workflow run 26877237491 failed (agent job, exit 1). Investigation found three compounding causes — one network/infra and two behavioral design flaws.

1. Network restriction — legacy AzDO feed host was firewall-blocked

The AWF firewall denied all 432 requests to dnceng.pkgs.visualstudio.com:443 (the only blocked domain in the run):

[WARN] Firewall blocked domains:
[WARN]   - Blocked: dnceng.pkgs.visualstudio.com:443
| dnceng.pkgs.visualstudio.com | 0 allowed | 432 denied |

NuGet.config splits feeds across two AzDO hosts. pkgs.dev.azure.com is covered by the dotnet ecosystem allowlist, but the dotnet8/dotnet9/dotnet10 feeds live on the legacy dnceng.pkgs.visualstudio.com host, which was not allowlisted. With the runtime-pack feed blocked, NuGet restore (and therefore the whole-repo build) failed.

2. Token-budget exhaustion: chasing the number of an issue it just created

The deeper root cause was not the build loop. After fast-failing the build, the agent created 4 tracking issues via create_issue, then spent 20+ tool calls trying to discover their issue numbers so it could write [ActiveIssue(.../issues/NNNN)] in the same run. That is architecturally impossible: create_issue is a gh-aw safe output filed by a post-run job — the tool returns only {"result":"success"} with no number during the agent run. The flailing ballooned context into the 25M effective-token hard rail:

attempt 1 failed: ... isMaxEffectiveTokensExceededError=true
done: exitCode=1 totalDuration=22m 5s

It was hard-railed before writing any safe output → no PR, exit 1. This wall is latent even on a green build, so the fast-fail fix below is necessary but not sufficient on its own.

3. Dedup-marker drift

The agent wrote <!-- flaky-test: … --> markers while every dedup search looks for <!-- flaky-test-id: … -->, and it re-filed pre-existing issue #13762 (which predates the marker convention).

Fix

• network.allowed: add dnceng.pkgs.visualstudio.com so restore can reach the dotnet8/9/10 feeds. Recompiled the lock (the firewall allowlist is frontmatter-derived).
• Step 6 (fail fast): a single environmental/network build failure goes straight to the Step 7d quarantine fallback instead of retrying or attempting a fix.
• Step 5 (two-run quarantine model): a flake is quarantine-eligible only if its tracking issue was already OPEN before this run with a readable number. A just-created issue does not count this run and becomes eligible next run. Explicitly forbids polling/guessing the number of an issue created this run — the exact behavior that exhausted the budget.
• Step 4 (dedup hardening): note that create_issue returns no number during the run; stress the marker must be copied exactly as <!-- flaky-test-id: … -->; add a title-search fallback for legacy issues lacking the marker; note the benign Malformed version: gh stderr warning so it is not misread as a failure.

The workflow body is runtime-imported ({{#runtime-import …}}), so the Step 4/5 prose edits take effect without regenerating the lock; only the network allowlist required a recompile.

---

Also in this PR: merge the detector skill into the workflow

The separate .github/skills/flaky-test-detector/SKILL.md duplicated guidance the workflow already states in its own steps; its only unique content (the evidence-source model + detector JSON field glosses) is now a compact Background section in flaky-test-detector.agent.md. The detector script moved to .github/workflows/scripts/Get-FlakyTests.ps1 (next to the workflow that calls it), and SKILL.md was deleted. Net −241 lines, one fewer moving part. No other workflow or pipeline referenced the skill/script; body is runtime-imported so no lock recompile is needed.

[Copilot]

Pull request overview

Updates the flaky-test-detector Agentic Workflow configuration to avoid restore/build failures caused by the legacy AzDO feed host being blocked by the firewall, and to prevent token-budget exhaustion by guiding the agent to quarantine rather than loop when the environment is broken.

Changes:

• Allowlist dnceng.pkgs.visualstudio.com in the workflow’s network.allowed so NuGet restore can reach the dotnet8/9/10 feeds referenced by NuGet.config.
• Add “fail fast” guidance to the build step so an initial environment/network build failure immediately triggers the Step 7d quarantine fallback instead of repeated rebuild attempts.
• Recompile the generated .agent.lock.yml so the firewall/allowed-domain metadata matches the updated frontmatter.

Show a summary per file

File	Description
.github/workflows/flaky-test-detector.agent.md	Adds the legacy feed host to the allowlist and documents a fail-fast path to quarantine when the initial build failure is environmental.
.github/workflows/flaky-test-detector.agent.lock.yml	Regenerates the lock so the runtime allowed-domain lists include dnceng.pkgs.visualstudio.com and prompt/config hashes reflect the updated source.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 0

[github-actions]

🔍 Skill Validator Results

⚠️ Warnings or advisories found

Scope	Checked
Skills	1
Agents	0
Total	1
Severity	Count
---	---:
❌ Errors	0
⚠️ Warnings	1
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	Found 1 skill(s)
ℹ️	[flaky-test-detector] 📊 flaky-test-detector: 3,897 BPE tokens [chars/4: 3,794] (standard ~), 13 sections, 5 code blocks
ℹ️	[flaky-test-detector] ⚠ Skill is 3,897 BPE tokens (chars/4 estimate: 3,794) — approaching "comprehensive" range where gains diminish.
ℹ️	✅ All checks passed (1 skill(s))

Full validator output

```text Found 1 skill(s) [flaky-test-detector] 📊 flaky-test-detector: 3,897 BPE tokens [chars/4: 3,794] (standard ~), 13 sections, 5 code blocks [flaky-test-detector] ⚠ Skill is 3,897 BPE tokens (chars/4 estimate: 3,794) — approaching "comprehensive" range where gains diminish. ✅ All checks passed (1 skill(s)) ```