Make flaky detector verify recurrence postdates the fix before commenting - #13930
Merged
Conversation
…ting The detector's closed-issue handling commented 'recurred' on any related issue closed within ~30 days, without checking whether the failures actually occurred after the fix merged. Because the look-back window (-DaysBack) routinely spans the fix date, stale pre-fix failures were mislabeled as recurrences (e.g. issue #13734, fixed the day before). Step 4 now requires confirming at least one failure's build start time is strictly after the fixing PR merged (using the closing PR's merge commit time, and build start times rather than the date-only lastSeen) before posting a recurrence comment; otherwise the evidence is treated as stale and the test is skipped for the run. SKILL.md gains a matching note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🔍 Skill Validator Results
Summary
Full validator output```text Found 1 skill(s) [flaky-test-detector] 📊 flaky-test-detector: 3,581 BPE tokens [chars/4: 3,475] (standard ~), 13 sections, 5 code blocks [flaky-test-detector] ⚠ Skill is 3,581 BPE tokens (chars/4 estimate: 3,475) — approaching "comprehensive" range where gains diminish. ✅ All checks passed (1 skill(s)) ``` |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the flaky-test detector’s guidance for handling recently-closed tracking issues so it only comments “recurred” when there is evidence of failures occurring after the fix PR merged, avoiding stale pre-fix failures being misclassified as recurrences.
Changes:
- Adds a timing-aware recurrence check for closed issues: require at least one post-fix failure (by build start time) before commenting.
- Documents the same rule in the flaky-test-detector skill’s “Interpreting Results” guidance.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/flaky-test-detector.agent.md | Adds post-fix timestamp verification steps before commenting on recently-closed issues. |
| .github/skills/flaky-test-detector/SKILL.md | Documents the post-fix recurrence rule in the skill guidance. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 1
gh issue view --json does not support a timelineItems field. Use the supported closedByPullRequestsReferences field to get the closing PR, then gh pr view --json mergedAt for its merge time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AlesProkop
approved these changes
Jun 3, 2026
Member
|
@ViktorHofer pls squash next time |
This was referenced Jun 4, 2026
This was referenced Aug 11, 2026
Bump Microsoft.Build from 18.4.0 to 18.9.6
SkylineCommunications/Skyline.DataMiner.CICD.Packages#172
Open
This was referenced Aug 12, 2026
Open
Merged
Merged
WarperSan
pushed a commit
to WarperSan/ThunderPipe
that referenced
this pull request
Aug 12, 2026
Updated [Microsoft.Build.Utilities.Core](https://github.com/dotnet/msbuild) from 18.8.2 to 18.9.6. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Build.Utilities.Core's releases](https://github.com/dotnet/msbuild/releases)._ ## 18.9.6 ## What's Changed * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13793 * [vs18.0] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13859 * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13858 * [vs18.7] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13863 * CsWin32 follow-up: CLR metadata + TypeLib interop migration by @JeremyKuhne in dotnet/msbuild#13853 * Update vmr-sb-validation.yml for Azure Pipelines by @meghnave in dotnet/msbuild#13871 * Test: keep shell alive 15s in ToolTaskCanChangeCanonicalErrorFormat (#13734) by @jankratochvilcz in dotnet/msbuild#13878 * Add vs18.8 to merge-flow config by @OvesN in dotnet/msbuild#13877 * Stable branding for 18.8 release by @OvesN in dotnet/msbuild#13883 * Bump main to 18.9.0 after vs18.8 snap by @OvesN in dotnet/msbuild#13880 * Avoid checkout in insertion pipeline by @rainersigwald in dotnet/msbuild#13887 * Report actual launch path in MSB4216 for Runtime="NET" task host by @ViktorHofer in dotnet/msbuild#13889 * Migrate Tlblmp and AxImp to Multithreaded Execution by @AlesProkop in dotnet/msbuild#13708 * Replace ErrorUtilities assertion methods with Assumed API and BCL throw helpers by @DustinCampbell in dotnet/msbuild#13790 * Fix CLR_E_SHIM_RUNTIMELOAD in RAR's IMetaDataDispenser activation by @JeremyKuhne in dotnet/msbuild#13899 * [main] Update dependencies from nuget/nuget.client by @dotnet-maestro[bot] in dotnet/msbuild#13905 * [main] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13907 * [main] Update dependencies from dotnet/roslyn by @dotnet-maestro[bot] in dotnet/msbuild#13910 * [vs17.14] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13908 * [vs18.0] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13906 * Fix ToolTask output loss: increase EOF pipe timeout from 2s to 30s by @huulinhnguyen-dev in dotnet/msbuild#13767 * Improve symlink cycle condition by @GangWang01 in dotnet/msbuild#13901 * [vs18.6] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13904 * Add flaky-test detection and auto-fix agentic workflows by @ViktorHofer in dotnet/msbuild#13915 * Quote --ignore-exit-code values so the quarantine pipeline does not shell-split on Unix by @ViktorHofer in dotnet/msbuild#13918 * Fix flaky-test detector PR-evidence loss and raise scan limits by @ViktorHofer in dotnet/msbuild#13919 * Tighten the pr review agent by @JanKrivanek in dotnet/msbuild#13921 * Add environment variables for governance detection by @ViktorHofer in dotnet/msbuild#13920 * Change IsPackable to true and add IsShipping flag by @ViktorHofer in dotnet/msbuild#13924 * [vs18.7] Update dependencies from dotnet/arcade by @dotnet-maestro[bot] in dotnet/msbuild#13911 * Make flaky detector verify recurrence postdates the fix before commenting by @ViktorHofer in dotnet/msbuild#13930 * Fix AbsolutePath.GetCanonicalForm process state leak on Windows by @OvesN in dotnet/msbuild#13788 * Fix flaky detector: unblock dnceng feed, fail fast, and defer quarantine to a second run by @ViktorHofer in dotnet/msbuild#13936 * Update documentation for ImplicitUsings element by @drewnoakes in dotnet/msbuild#13900 * [vs18.6] Point OptProf bootstrapper at rel/stable instead of int.main by @AlesProkop in dotnet/msbuild#13923 * Add CS8618 suppressor for required MSBuild task properties by @AArnott in dotnet/msbuild#13926 * Tighten NodeLaunchData.EnvironmentOverrides nullability to IDictionary<string, string?>? by @OvesN with @Copilot in dotnet/msbuild#13815 * Fix ToolTask EOF wait to be STA-safe via CountdownEvent (MSB4018 in AspNetCompiler) by @YuliiaKovalova in dotnet/msbuild#13917 * [automated] Merge branch 'vs18.6' => 'vs18.7' by @github-actions[bot] in dotnet/msbuild#13941 * Localized file check-in by OneLocBuild Task: Build definition ID 9434: Build ID 14192258 by @dotnet-bot in dotnet/msbuild#13849 * Bumping to 10.0.8 runtime packages by @OvesN in dotnet/msbuild#13898 * Flaky-test workflow: reassure on empty PR list + drop local reproduction (quarantine-first) by @ViktorHofer in dotnet/msbuild#13938 * [Flaky Test] Un-quarantine 5 consistently-green tests by @github-actions[bot] in dotnet/msbuild#13952 * Flaky-test detector: open PRs ready-for-review; drop newly-filed-issues section from PR body by @ViktorHofer in dotnet/msbuild#13958 * [Flaky Test] Quarantine 4 flaky tests by @github-actions[bot] in dotnet/msbuild#13937 * Flaky-test: fix duplicate-issue bug by switching dedup key to a visible code-block key by @ViktorHofer in dotnet/msbuild#13963 * CsWin32 follow-up: WindowsNative + VS Setup Configuration + remaining hand-rolled interop by @JeremyKuhne in dotnet/msbuild#13872 * Add the reviewer release skill checking if the Learn article Change waves is updated by @GangWang01 in dotnet/msbuild#13840 * [main] Source code updates from dotnet/dotnet by @dotnet-maestro[bot] in dotnet/msbuild#13977 ... (truncated) Commits viewable in [compare view](dotnet/msbuild@v18.8.2...v18.9.6). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Aug 12, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the flaky-test detector's closed-issue handling timing-aware so it stops mislabeling stale, pre-fix failures as recurrences.
Problem
When the detector finds a flaky test that maps to a recently-closed tracking issue, it commented "recurred" on that issue without checking when the failures occurred. Because the look-back window (
-DaysBack) routinely spans the date a fix merged, failures from before the fix were treated as evidence of recurrence. For example, issue #13734 was fixed via a merged PR the day before, yet the detector commented that it had recurred based on pre-fix builds.Change
.github/workflows/flaky-test-detector.agent.md(Step 4): before posting a recurrence comment on a closed issue, the agent must now confirm that at least one failure's build start time is strictly after the fixing PR merged (using the closing PR's merge-commit time and actual build start times, not the date-onlylastSeen). If every failure predates the fix/close, the evidence is treated as stale: no comment, no reopen, skip the test for the run..github/skills/flaky-test-detector/SKILL.md: companion note in the "Interpreting Results" section documenting the same rule.Body/prose-only edits to the agent workflow and skill — no lock recompile required.