Add evals to 24 more agentic workflows - #45984
Merged
Merged
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
pelikhan
July 16, 2026 12:16
View session
pelikhan
marked this pull request as ready for review
July 16, 2026 12:17
Contributor
There was a problem hiding this comment.
Pull request overview
Adds BinEval coverage to 24 agentic workflows, evaluating both primary actions and valid no-change outcomes. Five questions currently exclude valid workflow outcomes and require adjustment.
Changes:
- Adds two binary evaluation questions per workflow.
- Regenerates compiled workflows with evaluation and state-persistence jobs.
- Extends evaluation coverage from 14 to 38 workflows.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/aw-failure-investigator.md |
Adds failure-investigation evals. |
.github/workflows/aw-failure-investigator.lock.yml |
Compiles the new evals. |
.github/workflows/cli-consistency-checker.md |
Adds CLI-consistency evals. |
.github/workflows/cli-consistency-checker.lock.yml |
Compiles the new evals. |
.github/workflows/code-scanning-fixer.md |
Adds alert-fixing evals. |
.github/workflows/code-scanning-fixer.lock.yml |
Compiles the new evals. |
.github/workflows/commit-changes-analyzer.md |
Adds commit-analysis evals. |
.github/workflows/commit-changes-analyzer.lock.yml |
Compiles the new evals. |
.github/workflows/daily-compiler-quality.md |
Adds compiler-quality evals. |
.github/workflows/daily-compiler-quality.lock.yml |
Compiles the new evals. |
.github/workflows/daily-doc-updater.md |
Adds documentation-update evals. |
.github/workflows/daily-doc-updater.lock.yml |
Compiles the new evals. |
.github/workflows/daily-function-namer.md |
Adds function-naming evals. |
.github/workflows/daily-function-namer.lock.yml |
Compiles the new evals. |
.github/workflows/daily-malicious-code-scan.md |
Adds malicious-code scan evals. |
.github/workflows/daily-malicious-code-scan.lock.yml |
Compiles the new evals. |
.github/workflows/daily-semgrep-scan.md |
Adds Semgrep scan evals. |
.github/workflows/daily-semgrep-scan.lock.yml |
Compiles the new evals. |
.github/workflows/daily-syntax-error-quality.md |
Adds syntax-quality evals. |
.github/workflows/daily-syntax-error-quality.lock.yml |
Compiles the new evals. |
.github/workflows/daily-yamllint-fixer.md |
Adds YAML-lint evals. |
.github/workflows/daily-yamllint-fixer.lock.yml |
Compiles the new evals. |
.github/workflows/dependabot-go-checker.md |
Adds Dependabot compatibility evals. |
.github/workflows/dependabot-go-checker.lock.yml |
Compiles the new evals. |
.github/workflows/glossary-maintainer.md |
Adds glossary-maintenance evals. |
.github/workflows/glossary-maintainer.lock.yml |
Compiles the new evals. |
.github/workflows/issue-monster.md |
Adds issue-assignment evals. |
.github/workflows/issue-monster.lock.yml |
Compiles the new evals. |
.github/workflows/issue-triage-agent.md |
Adds issue-triage evals. |
.github/workflows/issue-triage-agent.lock.yml |
Compiles the new evals. |
.github/workflows/pr-code-quality-reviewer.md |
Adds PR-review evals. |
.github/workflows/pr-code-quality-reviewer.lock.yml |
Compiles the new evals. |
.github/workflows/refiner.md |
Adds code-refinement evals. |
.github/workflows/refiner.lock.yml |
Compiles the new evals. |
.github/workflows/security-review.md |
Adds security-review evals. |
.github/workflows/security-review.lock.yml |
Compiles the new evals. |
.github/workflows/spec-enforcer.md |
Adds specification-enforcement evals. |
.github/workflows/spec-enforcer.lock.yml |
Compiles the new evals. |
.github/workflows/stale-repo-identifier.md |
Adds stale-repository evals. |
.github/workflows/stale-repo-identifier.lock.yml |
Compiles the new evals. |
.github/workflows/sub-issue-closer.md |
Adds sub-issue closure evals. |
.github/workflows/sub-issue-closer.lock.yml |
Compiles the new evals. |
.github/workflows/technical-doc-writer.md |
Adds technical-documentation evals. |
.github/workflows/technical-doc-writer.lock.yml |
Compiles the new evals. |
.github/workflows/tidy.md |
Adds formatting and tidying evals. |
.github/workflows/tidy.lock.yml |
Compiles the new evals. |
.github/workflows/unbloat-docs.md |
Adds documentation-simplification evals. |
.github/workflows/unbloat-docs.lock.yml |
Compiles the new evals. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 48/48 changed files
- Comments generated: 4
- Review effort level: Medium
Comment on lines
+449
to
+450
| - id: single_issue_scoped | ||
| question: Does the agent output show that at most one issue was assigned to Copilot per run? |
Comment on lines
+45
to
+46
| - id: findings_specific | ||
| question: Does the agent output describe specific, concrete security concerns tied to the pull request changes rather than generic advice? |
Comment on lines
+324
to
+325
| - id: issues_created_or_closed | ||
| question: Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed? |
Comment on lines
+85
to
+86
| - id: pr_or_comment_created | ||
| question: Was a pull request or comment created with documentation improvements for the requested topic? |
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends BinEval coverage from 14 to 38 workflows by adding 2 binary eval questions to each of 24 workflows that already declare
safe-outputs(required foragent_output.jsonavailability).Pattern
Each pair follows the same structure: one question verifying the core action was taken, one verifying the noop/boundary condition was respected.
Workflows updated (24)
pr-code-quality-reviewer,refiner,tidy,duplicate-…(see below)security-review,daily-malicious-code-scan,daily-semgrep-scan,code-scanning-fixerdaily-doc-updater,glossary-maintainer,technical-doc-writer,unbloat-docsissue-triage-agent,issue-monster,sub-issue-closer,stale-repo-identifierdaily-yamllint-fixer,daily-syntax-error-quality,daily-function-namer,daily-compiler-quality,aw-failure-investigatordependabot-go-checker,spec-enforcercli-consistency-checker,commit-changes-analyzerExclusions
duplicate-code-detector,changeset, andnecromancer(allengine: codex) trigger a compiler bug — duplicateDownload container imagesstep in the generatedevalsjob. Excluded pending a compiler fix.