[evals] Daily Evals Feature Report - 2026-07-28

[github-actions]

Executive Summary

18 evals-enabled runs produced evals.jsonl for 15 workflows. The evals job itself is healthy at 100% success, but the judge output is not clean: 18 of 37 records are UNKNOWN, and the valid YES/NO records only reach a 57.9% YES rate.

Note

Status: DEGRADED - evals artifacts were produced for every evals-enabled run, but a large share of judge outputs are UNKNOWN and the valid YES rate is below the healthy threshold.

Key Metrics

Metric	Value
Workflows with evals	15
Runs analyzed	18
Runs with evals results	18
Evals job success rate	100.0%
Overall YES rate	57.9%
UNKNOWN answers	18

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
Auto-Triage Issues	1	100.0%	100.0%	labels-applied: "Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?" (100.0% YES)
CLI Version Checker	1	100.0%	0.0%	cli_versions_checked: "Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?" (0.0% YES)
Code Scanning Fixer	1	100.0%	0.0%	pr_created_or_noop: "Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?" (0.0% YES)
Contribution Check	1	100.0%	0.0%	prs-evaluated: "Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?" (0.0% YES)
Copilot Session Insights	1	100.0%	0.0%	insights_report_produced: "Was a report produced with usage patterns, success rates, and performance metrics?" (0.0% YES)
Daily Cli Tools Tester	1	100.0%	100.0%	cli_tools_tested: "Did the agent run exploratory tests on the audit, logs, and compile CLI tools?" (100.0% YES)
Daily Elixir Credo Snippet Audit	1	100.0%	0.0%	elixir_snippets_linted: "Did the agent use Credo to lint Elixir snippets found in repository markdown files?" (0.0% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	0.0%	conformance_checked: "Did the agent run a conformance check against the Safe Outputs specification implementation?" (0.0% YES)
Daily VulnHunter Scan	1	100.0%	0.0%	issue_created_or_noop: "Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?" (0.0% YES)
Issue Arborist	1	100.0%	0.0%	issues_analyzed: "Did the agent analyze recent issues and identify related issue relationships?" (0.0% YES)
Issue Monster	2	100.0%	0.0%	single_issue_scoped: "Does the agent output show that at most one issue was assigned to Copilot per run?" (0.0% YES)
PR Code Quality Reviewer	3	100.0%	0.0%	findings_scoped: "Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?" (0.0% YES)
Sub-Issue Closer	1	100.0%	100.0%	issues_checked: "Did the agent check parent issues for the completion status of all their sub-issues?" (100.0% YES)
Tidy	1	100.0%	0.0%	pr_created_or_noop: "Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?" (0.0% YES)
[aw] Failure Investigator (6h)	1	100.0%	0.0%	failures_investigated: "Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?" (0.0% YES)

Per-Question Breakdown per Workflow

Auto-Triage Issues

Question ID	Question	YES	NO	UNKNOWN	YES Rate
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	1	0	0	100.0%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	1	0	0	100.0%

CLI Version Checker

Question ID	Question	YES	NO	UNKNOWN	YES Rate
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	0	1	0	0.0%
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	0	0	1	0.0%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	0	0	1	0.0%

Code Scanning Fixer

Question ID	Question	YES	NO	UNKNOWN	YES Rate
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	1	0	0	100.0%
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	0	1	0.0%

Contribution Check

Question ID	Question	YES	NO	UNKNOWN	YES Rate
prs-evaluated	Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?	0	0	1	0.0%
report-created	Was a report issue created summarizing PR compliance with the contributing guidelines?	0	0	1	0.0%

Copilot Session Insights

Question ID	Question	YES	NO	UNKNOWN	YES Rate
insights_report_produced	Was a report produced with usage patterns, success rates, and performance metrics?	0	0	1	0.0%
sessions_analyzed	Did the agent analyze GitHub Copilot coding agent sessions?	0	1	0	0.0%

Daily Cli Tools Tester

Question ID	Question	YES	NO	UNKNOWN	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	1	0	0	100.0%
test_results_reported	Were the test results reported with any issues or anomalies identified?	1	0	0	100.0%

Daily Elixir Credo Snippet Audit

Question ID	Question	YES	NO	UNKNOWN	YES Rate
elixir_snippets_linted	Did the agent use Credo to lint Elixir snippets found in repository markdown files?	0	0	1	0.0%
fixes_proposed_or_noop	Were fixes proposed in a PR or issue, or was noop used when no linting violations were found?	0	0	1	0.0%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	UNKNOWN	YES Rate
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	0	1	0	0.0%
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	0	0	1	0.0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	UNKNOWN	YES Rate
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	0	0	1	0.0%
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?	0	0	1	0.0%

Issue Arborist

Question ID	Question	YES	NO	UNKNOWN	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	0	1	0.0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	0	1	0.0%

Issue Monster

Question ID	Question	YES	NO	UNKNOWN	YES Rate
issue_assigned	Did the agent assign at least one issue to the Copilot coding agent, or correctly skip when no suitable issues were found?	2	0	0	100.0%
single_issue_scoped	Does the agent output show that at most one issue was assigned to Copilot per run?	0	2	0	0.0%

PR Code Quality Reviewer

Question ID	Question	YES	NO	UNKNOWN	YES Rate
findings_scoped	Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?	0	0	3	0.0%
review_posted	Did the agent post a code review comment on the pull request?	2	1	0	66.7%

Sub-Issue Closer

Question ID	Question	YES	NO	UNKNOWN	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	0	100.0%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	0	100.0%

Tidy

Question ID	Question	YES	NO	UNKNOWN	YES Rate
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	0	0	1	0.0%
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	0	1	0	0.0%

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	UNKNOWN	YES Rate
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	0	1	0	0.0%
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	0	0	1	0.0%

Quality Signals

• single_issue_scoped: 100.0% NO across 2 binary judgments, with 0 UNKNOWN outputs; workflows: Issue Monster.
• cli_versions_checked: 100.0% NO across 1 binary judgments, with 0 UNKNOWN outputs; workflows: CLI Version Checker.
• conformance_checked: 100.0% NO across 1 binary judgments, with 0 UNKNOWN outputs; workflows: Daily Safe Outputs Conformance Checker.
• UNKNOWN outputs are concentrated in workflows that likely lack enough agent-output context for the judge or are producing non-binary parser output; this is the main reason the feature is not healthy.

Recommendations

• Investigate why the evals judge is emitting UNKNOWN instead of a binary answer, starting with workflows that have only UNKNOWN records.
• Tighten eval questions that routinely fail, especially the PR-review and issue-assignment workflows, so each question maps to a single observable signal.
• Check workflows that depend on agent output for evals coverage and make sure they declare the required context (safe-outputs / agent artifact) before expecting binary judgments.

References

• §30340496775
• §30339642461
• §30339510350

Generated by 🧪 Daily Evals Feature Report · gpt54 · 22 AIC · ⌖ 3.18 AIC · ⊞ 11.1K · ◷

• expires on Aug 4, 2026, 12:13 AM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #48821

View newer issue

---

This action was performed automatically by the Daily Evals Feature Report workflow.