[evals] Daily Evals Feature Report - 2026-07-22

[github-actions]

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Executive Summary

Analyzed the 40 most recent evals result runs across 6 workflows. Evals job delivery is stable, but answer quality is weak: only Auto-Triage Issues produced any passing runs, and the other workflows are emitting UNKNOWN responses that I counted as non-YES because the contract is binary.

Note

Status: DEGRADED - 100.0% evals-job success, but only 42.5% YES rate and most workflows are not returning binary YES/NO answers.

Key Metrics

Metric	Value
Workflows with evals	6
Runs analyzed	40
Runs with evals results	40
Evals job success rate	100.0%
Overall YES rate	42.5%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
Auto-Triage Issues	23	100.0%	47.8%	"Was a summary discussion created listing the issues processed and the labels applied?" (47.8% YES)
Avenger	5	100.0%	0.0%	"Did the agent assess the current CI state and determine if intervention was needed?" (0.0% YES)
Daily Documentation Healer	6	100.0%	0.0%	"Did the workflow identify at least one confirmed documentation gap to fix, or correctly conclude that no actionable gap remained?" (0.0% YES)
Daily Observability Report for AWF Firewall and MCP Gateway	2	100.0%	0.0%	"Did the agent analyze logging and telemetry coverage for AWF firewall and MCP Gateway?" (0.0% YES)
Daily Reliability Review	2	100.0%	0.0%	"Did the agent analyze agentic workflow failures and regressions using Sentry traces?" (0.0% YES)
Daily Security Red Team	2	100.0%	0.0%	"Did the agent perform a deep red teaming security scan of actions/setup/js and actions/setup/sh directories?" (0.0% YES)

Per-Question Breakdown per Workflow

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	23	0	100%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	11	12	47.8%

Avenger

Question ID	Question	YES	NO	YES Rate
ci_state_assessed	Did the agent assess the current CI state and determine if intervention was needed?	0	5	0%
pr_created_or_skipped	Was a PR created with CI fixes, or was the run correctly skipped because CI was already passing?	0	5	0%

Daily Documentation Healer

Question ID	Question	YES	NO	YES Rate
gaps-confirmed	Did the workflow identify at least one confirmed documentation gap to fix, or correctly conclude that no actionable gap remained?	0	6	0%
pr-issue-or-noop	Was a documentation pull request or issue created for confirmed gaps, or was noop used appropriately when nothing required action?	0	6	0%

Daily Observability Report for AWF Firewall and MCP Gateway

Question ID	Question	YES	NO	YES Rate
observability_data_analyzed	Did the agent analyze logging and telemetry coverage for AWF firewall and MCP Gateway?	0	2	0%
report_created	Was an observability report created identifying gaps and trends?	0	2	0%

Daily Reliability Review

Question ID	Question	YES	NO	YES Rate
failures_analyzed	Did the agent analyze agentic workflow failures and regressions using Sentry traces?	0	2	0%
reliability_report_created	Was a daily reliability review report created with actionable findings?	0	2	0%

Daily Security Red Team

Question ID	Question	YES	NO	YES Rate
security_scan_performed	Did the agent perform a deep red teaming security scan of actions/setup/js and actions/setup/sh directories?	0	2	0%
findings_reported_or_noop	Were security findings (backdoors, secret leaks, malicious code) reported, or was noop used when no issues were detected?	0	2	0%

Quality Signals

• The evals judges are not emitting binary outcomes for 5 of 6 workflows; UNKNOWN should be treated as a blocking regression in the evals contract.
• Auto-Triage Issues is the only workflow with a mixed signal; report-created is still below 50% YES.
• The other workflows are flat 0% YES on both questions, which suggests the prompts or the judge rubric are not aligned with the expected behavior.

Recommendations

• Fix the evals output path so judges emit YES/NO instead of UNKNOWN for the affected workflows.
• Tighten the prompts or scoring rubric for Avenger, Daily Security Red Team, and Daily Reliability Review first, since they are at 0% YES across the board.
• Rework the Auto-Triage Issues report-created prompt; it is the only question with a partial pass rate and it remains the weakest signal in the sample.

References

• §29898962758
• §29878874785
• §29876243095

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 Daily Evals Feature Report · gpt54 83.5 AIC · ⊞ 29.8K · ◷

• expires on Jul 29, 2026, 12:23 AM UTC-08:00