Improve CI scanner duplicate KBE detection - #131312
Conversation
Use the Build Analysis GitHub check as a positive-only dedup signal and search complete test identifiers before shortened stems. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d555b634-18ce-458d-a831-096e0b0c87b3
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Updates the CI Outer-Loop Failure Scanner guidance to reduce duplicate Known Build Error (KBE) filings by improving how it searches for existing KBEs and by leveraging Build Analysis results exposed via a GitHub check instead of an AzDO attachment.
Changes:
- Add guidance to search full test method identifiers verbatim (especially underscore-delimited names) before deriving shorter stems.
- Update the scanner’s “Build Analysis” lookup instructions to use the GitHub check-runs API (instead of a referenced-but-nonexistent AzDO attachment).
- Refresh the generated workflow lock metadata hash to reflect the updated source workflow text.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/workflows/shared/create-kbe.instructions.md |
Adds guidance to avoid truncating underscore-delimited test identifiers during existing-KBE searches. |
.github/workflows/ci-failure-scan.md |
Replaces Build Analysis “attachment” guidance with a best-effort lookup via the Build Analysis GitHub check. |
.github/workflows/ci-failure-scan.lock.yml |
Updates gh-aw metadata (body_hash) to match the edited workflow source. |
kotlarmilos
left a comment
There was a problem hiding this comment.
One minor comment, otherwise LGTM!
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
.github/workflows/shared/create-kbe.instructions.md:68
- The semicolon after "underscore-delimited identifiers" reads like an accidental sentence break; using a period here improves grammar and avoids implying an additional clause.
When a failure includes a complete test method identifier, search that
identifier verbatim before deriving any shorter stem. Do not truncate
underscore-delimited identifiers;
GitHub search does not reliably prefix-match them. Only strip the specific
platform, architecture, and type-width suffixes described in variation 6.
Why
The CI failure scanner filed #130940 even though #129691 already covered the same failure. The run searched a shortened underscore-delimited method name, which GitHub did not match as a prefix of the complete test identifier.
The scanner instructions also claimed that Build Analysis results were available through an AzDO attachment named
Build_Analysis_KnownIssues_v1. Investigation found no producer or observed instance of that attachment. Build Analysis did recognize the failure, but that information was exposed through its GitHub check instead.Changes
Build AnalysisGitHub check.Validation
A targeted dry-mode eval replayed build 1505119. The rendered Build Analysis check omitted #129691, then the verbatim full-method search found it and produced
existing-kbe #129691with no issue creation. All 8 targeted graders passed.cc: @kotlarmilos
Note
This pull request description was generated with GitHub Copilot.