[evals] Daily Evals Feature Report - 2026-07-31

[github-actions]

Executive Summary

Across 28 workflows and 34 runs, the evals feature is DEGRADED: the eval job itself is healthy, but the overall YES rate is only 48.6%. That low YES rate is driven by 38 non-YES answers, including 32 UNKNOWN labels and 6 explicit NOs.

Note

Status: DEGRADED - 100.0% eval-job success and 48.6% overall YES rate.

Key Metrics

Metric	Value
Workflows with evals	28
Runs analyzed	34
Runs with evals results	34
Evals job success rate	100.0%
Overall YES rate	48.6%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
[aw] Failure Investigator (6h)	1	100.0%	0.0%	"Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?" (0.0% YES)
AI Moderator	1	100.0%	0.0%	"Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop?" (0.0% YES)
Auto-Triage Issues	1	100.0%	100.0%	"Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?" (100.0% YES)
CLI Version Checker	1	100.0%	0.0%	"Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?" (0.0% YES)
Code Scanning Fixer	1	100.0%	0.0%	"Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?" (0.0% YES)
Contribution Check	1	100.0%	0.0%	"Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?" (0.0% YES)
Copilot Session Insights	1	100.0%	0.0%	"Was a report produced with usage patterns, success rates, and performance metrics?" (0.0% YES)
Daily AgentRx Trace Optimizer	1	100.0%	0.0%	"Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?" (0.0% YES)
Daily Assign Issue To User	1	100.0%	0.0%	"Did the agent post a comment explaining the assignment decision?" (0.0% YES)
Daily Cli Tools Tester	2	100.0%	100.0%	"Did the agent run exploratory tests on the audit, logs, and compile CLI tools?" (100.0% YES)
Daily Container Image Security Scan	1	100.0%	100.0%	"Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?" (100.0% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	0.0%	"Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?" (0.0% YES)
Daily VulnHunter Scan	1	100.0%	0.0%	"Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?" (0.0% YES)
Daily Windows Terminal Integration Builder	1	100.0%	100.0%	"Did the agent create an issue for an actionable integration failure, or use noop when no action was required?" (100.0% YES)
Daily Workflow Updater	1	100.0%	0.0%	"Did the agent create a pull request for required updates, or report that no changes were needed?" (0.0% YES)
Dependabot Burner	1	100.0%	0.0%	"Did the agent analyze the selected grouped Dependabot remediation batch?" (0.0% YES)
Design Decision Gate 🏗️	2	100.0%	0.0%	"Did the agent add a PR comment, push a draft ADR, or call noop?" (0.0% YES)
Discussion Task Miner - Code Quality Improvement Agent	1	100.0%	0.0%	"Does the agent output confirm that the created issues include the expected labels (code-quality, automation, task-mining)?" (0.0% YES)
ESLint Refiner	1	100.0%	0.0%	"Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?" (0.0% YES)
Issue Arborist	1	100.0%	0.0%	"Did the agent analyze recent issues and identify related issue relationships?" (0.0% YES)
Issue Monster	2	100.0%	0.0%	"Does the agent output show that at most one issue was assigned to Copilot per run?" (0.0% YES)
Multi-Device Docs Tester	1	100.0%	100.0%	"Did the agent test the documentation site across the requested device form factors?" (100.0% YES)
PR Code Quality Reviewer	2	100.0%	0.0%	"Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?" (0.0% YES)
PR Sous Chef	2	100.0%	100.0%	"Did the agent add a comment to at least one pull request?" (100.0% YES)
PR Triage Agent	1	100.0%	0.0%	"Does the agent output include a triage report summarizing the PRs processed?" (0.0% YES)
Sub-Issue Closer	1	100.0%	100.0%	"Did the agent check parent issues for the completion status of all their sub-issues?" (100.0% YES)
Test Quality Sentinel	2	100.0%	0.0%	"Does the agent output show that the objective for experiment model_size was successfully completed?" (0.0% YES)
Tidy	1	100.0%	100.0%	"Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?" (100.0% YES)

Per-Question Breakdown per Workflow

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	YES Rate
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	0	1	0.0%
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	1	0	100.0%

AI Moderator

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop?	0	1	0.0%
no-unsupported-action	Does the agent output show that only allowed safe-output actions (add-labels, hide-comment, noop) were used?	0	1	0.0%
rationale-provided	Does the agent output include a rationale explaining why the label(s) were applied or why noop was called?	0	1	0.0%

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	1	0	100.0%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	1	0	100.0%

CLI Version Checker

Question ID	Question	YES	NO	YES Rate
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	0	1	0.0%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	0	1	0.0%
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	1	0	100.0%

Code Scanning Fixer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	1	0.0%
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	1	0	100.0%

Contribution Check

Question ID	Question	YES	NO	YES Rate
prs-evaluated	Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?	0	1	0.0%
report-created	Was a report issue created summarizing PR compliance with the contributing guidelines?	0	1	0.0%

Copilot Session Insights

Question ID	Question	YES	NO	YES Rate
insights_report_produced	Was a report produced with usage patterns, success rates, and performance metrics?	0	1	0.0%
sessions_analyzed	Did the agent analyze GitHub Copilot coding agent sessions?	1	0	100.0%

Daily AgentRx Trace Optimizer

Question ID	Question	YES	NO	YES Rate
sub_agent_strategy_goal_met	Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?	0	1	0.0%

Daily Assign Issue To User

Question ID	Question	YES	NO	YES Rate
comment-posted	Did the agent post a comment explaining the assignment decision?	0	1	0.0%
contributor-selected	Does the agent output show that a relevant contributor was selected from recent merged PRs?	0	1	0.0%
assignment-made	Did the agent assign an issue to a user?	1	0	100.0%

Daily Cli Tools Tester

Question ID	Question	YES	NO	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	2	0	100.0%
test_results_reported	Were the test results reported with any issues or anomalies identified?	2	0	100.0%

Daily Container Image Security Scan

Question ID	Question	YES	NO	YES Rate
container_images_scanned	Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?	1	0	100.0%
findings_reported_or_noop	Did the agent report actionable image findings, or use noop when no findings required action?	1	0	100.0%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	YES Rate
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	0	1	0.0%
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	1	0	100.0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	YES Rate
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	0	1	0.0%
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?	0	1	0.0%

Daily Windows Terminal Integration Builder

Question ID	Question	YES	NO	YES Rate
failure_reported_or_noop	Did the agent create an issue for an actionable integration failure, or use noop when no action was required?	1	0	100.0%
windows_integration_checked	Did the agent assess the Windows CLI integration build and test workflow?	1	0	100.0%

Daily Workflow Updater

Question ID	Question	YES	NO	YES Rate
update_pr_created_or_noop	Did the agent create a pull request for required updates, or report that no changes were needed?	0	1	0.0%
action_versions_checked	Did the agent check GitHub Actions versions for available updates?	1	0	100.0%

Dependabot Burner

Question ID	Question	YES	NO	YES Rate
dependabot_batch_analyzed	Did the agent analyze the selected grouped Dependabot remediation batch?	0	1	0.0%
remediation_reported	Did the agent create a remediation pull request or clearly report why no remediation was needed?	1	0	100.0%

Design Decision Gate 🏗️

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent add a PR comment, push a draft ADR, or call noop?	0	2	0.0%
adr-check-performed	Does the agent output confirm that it checked for existing ADRs before deciding on an action?	0	2	0.0%
decision-justified	Does the agent output explain why an ADR is required or why no ADR gate was triggered for this PR?	0	2	0.0%

Discussion Task Miner - Code Quality Improvement Agent

Question ID	Question	YES	NO	YES Rate
labels-applied	Does the agent output confirm that the created issues include the expected labels (code-quality, automation, task-mining)?	0	1	0.0%
output-produced	Did the agent create at least one code quality issue or add a comment?	0	1	0.0%
tasks-extracted	Does the agent output show that actionable tasks were identified from the analyzed discussions?	0	1	0.0%

ESLint Refiner

Question ID	Question	YES	NO	YES Rate
refinements_reported	Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?	0	1	0.0%
eslint_trends_analyzed	Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?	1	0	100.0%

Issue Arborist

Question ID	Question	YES	NO	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	1	0.0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	1	0.0%

Issue Monster

Question ID	Question	YES	NO	YES Rate
single_issue_scoped	Does the agent output show that at most one issue was assigned to Copilot per run?	0	2	0.0%
issue_assigned	Did the agent assign at least one issue to the Copilot coding agent, or correctly skip when no suitable issues were found?	2	0	100.0%

Multi-Device Docs Tester

Question ID	Question	YES	NO	YES Rate
device_tests_completed	Did the agent test the documentation site across the requested device form factors?	1	0	100.0%
results_reported	Did the agent report the multi-device test results and any responsive design or functionality findings?	1	0	100.0%

PR Code Quality Reviewer

Question ID	Question	YES	NO	YES Rate
findings_scoped	Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?	0	2	0.0%
review_posted	Did the agent post a code review comment on the pull request?	2	0	100.0%

PR Sous Chef

Question ID	Question	YES	NO	YES Rate
comment-added	Did the agent add a comment to at least one pull request?	2	0	100.0%
nudge-targeted	Does the agent output show a specific reason why the selected PR needs a nudge toward maintainer investigation?	2	0	100.0%
pr-evaluated	Does the agent output confirm that it evaluated at least one open PR for nudge eligibility?	2	0	100.0%

PR Triage Agent

Question ID	Question	YES	NO	YES Rate
report-produced	Does the agent output include a triage report summarizing the PRs processed?	0	1	0.0%
triage-data-set	Does the agent output confirm that category, risk, and action data were determined for each processed PR?	0	1	0.0%
labels-applied	Did the agent apply triage labels to at least one pull request?	1	0	100.0%

Sub-Issue Closer

Question ID	Question	YES	NO	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	100.0%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	100.0%

Test Quality Sentinel

Question ID	Question	YES	NO	YES Rate
model_size_goal_met	Does the agent output show that the objective for experiment model_size was successfully completed?	0	2	0.0%

Tidy

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	1	0	100.0%
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	1	0	100.0%

Quality Signals

• 38 of 74 answers were non-YES, including 32 UNKNOWN labels, so the judge is often unable to emit a binary outcome.
• action-taken: Did the agent add a PR comment, push a draft ADR, or call noop? - 0.0% YES across 2 workflow(s): AI Moderator, Design Decision Gate 🏗️
• adr-check-performed: Does the agent output confirm that it checked for existing ADRs before deciding on an action? - 0.0% YES across 1 workflow(s): Design Decision Gate 🏗️
• comment-posted: Did the agent post a comment explaining the assignment decision? - 0.0% YES across 1 workflow(s): Daily Assign Issue To User

Recommendations

• Tighten prompts for the workflows with 0% YES on their weakest question, especially Design Decision Gate 🏗️, AI Moderator, and Daily Assign Issue To User.
• Reduce UNKNOWN outcomes by making the eval criteria more explicit or by adding clearer evidence requirements for the judge.
• Review the low-scoring prompt paths where the agent is expected to explain or justify an action rather than simply perform it.

References

• §30608805370
• §30610527224
• §30613210712

Generated by 🧪 Daily Evals Feature Report · gpt54 · 34.9 AIC · ⌖ 4.74 AIC · ⊞ 11.7K · ◷

• expires on Aug 7, 2026, 12:22 AM UTC-08:00