Skip to content

Improve CI scanner duplicate KBE detection - #131312

Merged
svick merged 2 commits into
dotnet:mainfrom
svick:ci-scan-full-test-name-dedup
Jul 28, 2026
Merged

Improve CI scanner duplicate KBE detection#131312
svick merged 2 commits into
dotnet:mainfrom
svick:ci-scan-full-test-name-dedup

Conversation

@svick

@svick svick commented Jul 24, 2026

Copy link
Copy Markdown
Member

Why

The CI failure scanner filed #130940 even though #129691 already covered the same failure. The run searched a shortened underscore-delimited method name, which GitHub did not match as a prefix of the complete test identifier.

The scanner instructions also claimed that Build Analysis results were available through an AzDO attachment named Build_Analysis_KnownIssues_v1. Investigation found no producer or observed instance of that attachment. Build Analysis did recognize the failure, but that information was exposed through its GitHub check instead.

Changes

  • Search complete test method identifiers verbatim before deriving shorter family stems.
  • Replace the nonexistent AzDO attachment with a best-effort lookup of the Build Analysis GitHub check.
  • Treat links found in the check as positive dedup evidence, while continuing exact KBE searches after a miss because large check reports are truncated.

Validation

A targeted dry-mode eval replayed build 1505119. The rendered Build Analysis check omitted #129691, then the verbatim full-method search found it and produced existing-kbe #129691 with no issue creation. All 8 targeted graders passed.

cc: @kotlarmilos

Note

This pull request description was generated with GitHub Copilot.

Use the Build Analysis GitHub check as a positive-only dedup signal and search complete test identifiers before shortened stems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d555b634-18ce-458d-a831-096e0b0c87b3
Copilot AI review requested due to automatic review settings July 24, 2026 12:04
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

@svick
svick marked this pull request as ready for review July 24, 2026 12:06
@svick
svick requested review from a team and jeffhandley as code owners July 24, 2026 12:06
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@svick
svick requested a review from kotlarmilos July 24, 2026 12:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the CI Outer-Loop Failure Scanner guidance to reduce duplicate Known Build Error (KBE) filings by improving how it searches for existing KBEs and by leveraging Build Analysis results exposed via a GitHub check instead of an AzDO attachment.

Changes:

  • Add guidance to search full test method identifiers verbatim (especially underscore-delimited names) before deriving shorter stems.
  • Update the scanner’s “Build Analysis” lookup instructions to use the GitHub check-runs API (instead of a referenced-but-nonexistent AzDO attachment).
  • Refresh the generated workflow lock metadata hash to reflect the updated source workflow text.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/shared/create-kbe.instructions.md Adds guidance to avoid truncating underscore-delimited test identifiers during existing-KBE searches.
.github/workflows/ci-failure-scan.md Replaces Build Analysis “attachment” guidance with a best-effort lookup via the Build Analysis GitHub check.
.github/workflows/ci-failure-scan.lock.yml Updates gh-aw metadata (body_hash) to match the edited workflow source.

Comment thread .github/workflows/ci-failure-scan.md

@kotlarmilos kotlarmilos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor comment, otherwise LGTM!

Comment thread .github/workflows/shared/create-kbe.instructions.md Outdated
Copilot AI review requested due to automatic review settings July 28, 2026 14:18
@svick
svick enabled auto-merge (squash) July 28, 2026 14:19
@svick
svick requested a review from kotlarmilos July 28, 2026 14:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/workflows/shared/create-kbe.instructions.md:68

  • The semicolon after "underscore-delimited identifiers" reads like an accidental sentence break; using a period here improves grammar and avoids implying an additional clause.
When a failure includes a complete test method identifier, search that
identifier verbatim before deriving any shorter stem. Do not truncate
underscore-delimited identifiers;
GitHub search does not reliably prefix-match them. Only strip the specific
platform, architecture, and type-width suffixes described in variation 6.

@svick
svick merged commit d025911 into dotnet:main Jul 28, 2026
22 checks passed
@svick
svick deleted the ci-scan-full-test-name-dedup branch July 29, 2026 09:27
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 11.0-rc1 milestone Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants