Flaky-test detector: open PRs ready-for-review; drop newly-filed-issues section from PR body - #13958
Merged
Merged
Conversation
…wly-filed-issues section from PR body Open the combined quarantine/un-quarantine PR as non-draft (draft: false) so dnceng CI validation runs immediately instead of only after a maintainer marks it ready. Also instruct the agent not to list create_issue-filed tracking issues in the PR body, since those issues are not acted on by the PR and the #number references create a misleading issue<->PR cross-link. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the flaky-test detector agentic workflow guidance and safe-output configuration so that its generated quarantine/un-quarantine PRs trigger CI immediately and its PR body only references issues that the PR actually acts on.
Changes:
- Switch combined quarantine/un-quarantine PR creation from draft to non-draft (ready-for-review) so build validation runs immediately.
- Update Step 8 PR-body guidance to avoid listing newly-filed tracking issues that aren’t part of the PR’s quarantine/un-quarantine edits.
- Regenerate the gh-aw lock file to reflect the frontmatter changes (including
draft: false) while preserving the existing schedule.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/flaky-test-detector.agent.md | Updates workflow description, PR creation mode, and PR-body instructions to avoid misleading issue↔PR cross-links. |
| .github/workflows/flaky-test-detector.agent.lock.yml | Regenerated lock reflecting updated frontmatter (notably draft: false) and updated embedded description text. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 0
…dy-for-review Detector: render each rolling build id in tracking-issue bodies (and OPEN-issue comments) as a markdown link to its AzDO build results page, deriving the URL from sampleBuildUrl's form, instead of plain-text ids. Fixer: open fix PRs as non-draft (draft: false) so dnceng CI validation runs immediately, matching the detector change. Lock regenerated; cron preserved at 38 12. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The directory-glob allowlist (src/**/*UnitTests*/**/*.cs) produced false negatives on valid test files (e.g. src/Build.UnitTests/Evaluation/ Preprocessor_Tests.cs), silently blocking legitimate fixes and failing the safe_outputs job. The product-code guard is instead provided by excluded-files + protected-files, the body's 'edit only the test's own file' rule plus a git diff check, and mandatory human review before any PR can merge. Lock regenerated; cron preserved at 38 12, draft:false retained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ViktorHofer
enabled auto-merge (squash)
June 5, 2026 12:32
JanKrivanek
approved these changes
Jun 5, 2026
This was referenced Jun 6, 2026
This was referenced Aug 11, 2026
Bump Microsoft.Build from 18.4.0 to 18.9.6
SkylineCommunications/Skyline.DataMiner.CICD.Packages#172
Open
This was referenced Aug 12, 2026
Open
Merged
Merged
WarperSan
pushed a commit
to WarperSan/ThunderPipe
that referenced
this pull request
Aug 12, 2026
Updated [Microsoft.Build.Utilities.Core](https://github.com/dotnet/msbuild) from 18.8.2 to 18.9.6. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Build.Utilities.Core's releases](https://github.com/dotnet/msbuild/releases)._ ## 18.9.6 ## What's Changed * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13793 * [vs18.0] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13859 * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13858 * [vs18.7] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13863 * CsWin32 follow-up: CLR metadata + TypeLib interop migration by @JeremyKuhne in dotnet/msbuild#13853 * Update vmr-sb-validation.yml for Azure Pipelines by @meghnave in dotnet/msbuild#13871 * Test: keep shell alive 15s in ToolTaskCanChangeCanonicalErrorFormat (#13734) by @jankratochvilcz in dotnet/msbuild#13878 * Add vs18.8 to merge-flow config by @OvesN in dotnet/msbuild#13877 * Stable branding for 18.8 release by @OvesN in dotnet/msbuild#13883 * Bump main to 18.9.0 after vs18.8 snap by @OvesN in dotnet/msbuild#13880 * Avoid checkout in insertion pipeline by @rainersigwald in dotnet/msbuild#13887 * Report actual launch path in MSB4216 for Runtime="NET" task host by @ViktorHofer in dotnet/msbuild#13889 * Migrate Tlblmp and AxImp to Multithreaded Execution by @AlesProkop in dotnet/msbuild#13708 * Replace ErrorUtilities assertion methods with Assumed API and BCL throw helpers by @DustinCampbell in dotnet/msbuild#13790 * Fix CLR_E_SHIM_RUNTIMELOAD in RAR's IMetaDataDispenser activation by @JeremyKuhne in dotnet/msbuild#13899 * [main] Update dependencies from nuget/nuget.client by @dotnet-maestro[bot] in dotnet/msbuild#13905 * [main] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13907 * [main] Update dependencies from dotnet/roslyn by @dotnet-maestro[bot] in dotnet/msbuild#13910 * [vs17.14] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13908 * [vs18.0] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13906 * Fix ToolTask output loss: increase EOF pipe timeout from 2s to 30s by @huulinhnguyen-dev in dotnet/msbuild#13767 * Improve symlink cycle condition by @GangWang01 in dotnet/msbuild#13901 * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13904 * Add flaky-test detection and auto-fix agentic workflows by @ViktorHofer in dotnet/msbuild#13915 * Quote --ignore-exit-code values so the quarantine pipeline does not shell-split on Unix by @ViktorHofer in dotnet/msbuild#13918 * Fix flaky-test detector PR-evidence loss and raise scan limits by @ViktorHofer in dotnet/msbuild#13919 * Tighten the pr review agent by @JanKrivanek in dotnet/msbuild#13921 * Add environment variables for governance detection by @ViktorHofer in dotnet/msbuild#13920 * Change IsPackable to true and add IsShipping flag by @ViktorHofer in dotnet/msbuild#13924 * [vs18.7] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13911 * Make flaky detector verify recurrence postdates the fix before commenting by @ViktorHofer in dotnet/msbuild#13930 * Fix AbsolutePath.GetCanonicalForm process state leak on Windows by @OvesN in dotnet/msbuild#13788 * Fix flaky detector: unblock dnceng feed, fail fast, and defer quarantine to a second run by @ViktorHofer in dotnet/msbuild#13936 * Update documentation for ImplicitUsings element by @drewnoakes in dotnet/msbuild#13900 * [vs18.6] Point OptProf bootstrapper at rel/stable instead of int.main by @AlesProkop in dotnet/msbuild#13923 * Add CS8618 suppressor for required MSBuild task properties by @AArnott in dotnet/msbuild#13926 * Tighten NodeLaunchData.EnvironmentOverrides nullability to IDictionary<string, string?>? by @OvesN with @Copilot in dotnet/msbuild#13815 * Fix ToolTask EOF wait to be STA-safe via CountdownEvent (MSB4018 in AspNetCompiler) by @YuliiaKovalova in dotnet/msbuild#13917 * [automated] Merge branch 'vs18.6' => 'vs18.7' by @github-actions[bot] in dotnet/msbuild#13941 * Localized file check-in by OneLocBuild Task: Build definition ID 9434: Build ID 14192258 by @dotnet-bot in dotnet/msbuild#13849 * Bumping to 10.0.8 runtime packages by @OvesN in dotnet/msbuild#13898 * Flaky-test workflow: reassure on empty PR list + drop local reproduction (quarantine-first) by @ViktorHofer in dotnet/msbuild#13938 * [Flaky Test] Un-quarantine 5 consistently-green tests by @github-actions[bot] in dotnet/msbuild#13952 * Flaky-test detector: open PRs ready-for-review; drop newly-filed-issues section from PR body by @ViktorHofer in dotnet/msbuild#13958 * [Flaky Test] Quarantine 4 flaky tests by @github-actions[bot] in dotnet/msbuild#13937 * Flaky-test: fix duplicate-issue bug by switching dedup key to a visible code-block key by @ViktorHofer in dotnet/msbuild#13963 * CsWin32 follow-up: WindowsNative + VS Setup Configuration + remaining hand-rolled interop by @JeremyKuhne in dotnet/msbuild#13872 * Add the reviewer release skill checking if the Learn article Change waves is updated by @GangWang01 in dotnet/msbuild#13840 * [main] Source code updates from dotnet/dotnet by @dotnet-maestro[bot] in dotnet/msbuild#13977 ... (truncated) Commits viewable in [compare view](dotnet/msbuild@v18.8.2...v18.9.6). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Aug 12, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #13938, addressing two pieces of feedback on how the flaky-test detector opens its combined quarantine/un-quarantine PR.
1. Open quarantine/un-quarantine PRs ready-for-review (not draft)
The detector previously opened its PR as a draft (
draft: true). dnceng AzDO build validation does not run on draft PRs, so CI only kicked in once a maintainer marked the PR ready for review (observed on #13952). Since these PRs are mechanical[ActiveIssue]add/removes that benefit from immediate CI validation, switch todraft: falseso CI runs as soon as the PR is opened.2. Don't list newly-filed tracking issues in the PR body
On an un-quarantine PR (#13952), the agent appended a "New flaky test issues filed this run" section referencing freshly-filed issues (e.g. #13949-#13951). Those issues are not acted on by the PR (they only become quarantine-eligible on a later run), so the
#numberreferences created a misleading issue<->PR cross-link. Step 8 now explicitly instructs the agent to reference only the issues for tests it actually quarantines or un-quarantines in that PR.Frontmatter changed (
draft+description), so the lock file was regenerated; cron is preserved at38 11.