Skip to content

Make threat-detect binary install step continue-on-error in warn mode - #52400

Open
pelikhan with Copilot wants to merge 3 commits into
mainfrom
copilot/aw-failures-harden-binary-installers
Open

Make threat-detect binary install step continue-on-error in warn mode#52400
pelikhan with Copilot wants to merge 3 commits into
mainfrom
copilot/aw-failures-harden-binary-installers

Conversation

Copilot AI commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

A GitHub Releases CDN blip caused the evals/detection job installers for the AWF firewall and threat-detect binaries to fail across multiple PR-gate workflows within a 25-minute window. The threat-detect binary download already had retry-with-backoff (curl --retry 5 --retry-delay 10 --retry-max-time 180), but a hard failure after retries still marked the whole detection job as failure — even though the job's own logic (GH_AW_DETECTION_CONTINUE_ON_ERROR) already treats a missing threat-detect binary as non-fatal in warn mode.

Compiler change

  • buildInstallThreatDetectStep now takes *WorkflowData and emits continue-on-error: true (or the configured expression) for the "Install threat-detect binary" step, reusing the same continue-on-error resolution already applied to the detection conclusion/analysis steps:
- name: Install threat-detect binary
  if: always() && steps.detection_guard.outputs.run_detection == 'true'
  continue-on-error: true
  run: |
    bash "${RUNNER_TEMP}/gh-aw/actions/install_threat_detect_binary.sh" latest

This keeps the install step's failure semantics consistent with the job's existing tolerance policy: in warn mode a transient download failure no longer flips the job conclusion to failure; in strict mode (continue-on-error: false) the step still fails the job as before.

Test coverage

  • threat_detection_isolation_test.go: added an assertion in TestExternalDetectorPath verifying the install step carries continue-on-error: true in the default warn mode.

Generated workflows

  • Recompiled all .lock.yml files; 104 workflows using features: gh-aw-detection: true pick up the new continue-on-error: true line on this step.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 26 AIC · ⌖ 5.66 AIC · ⊞ 8.5K ·
Comment /souschef to run again

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add retry-with-backoff to AWF/threat-detect binary installers Make threat-detect binary install step continue-on-error in warn mode Aug 13, 2026
Copilot AI requested a review from pelikhan August 13, 2026 03:20
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore | Risk: low | Priority: low | Score: 31/100

Score breakdown: impact 15/50, urgency 8/30, quality 8/20

CI: unknown (draft)

Recommended action: defer

CI hardening chore, still draft/blocked.

Automated triage — run 52400

Generated by 🔧 PR Triage Agent · auto · 67.2 AIC · ⌖ 2.77 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore · Risk: low · Priority: low
Score: 30/100 (impact 10, urgency 8, quality 12)
Recommended action: defer

Makes threat-detect binary install step continue-on-error in warn mode. Mostly bulk lock.yml regeneration from a one-line source change; low value/urgency.

Generated by 🔧 PR Triage Agent · auto · 61.7 AIC · ⌖ 2.63 AIC · ⊞ 7.8K ·

@pelikhan
pelikhan marked this pull request as ready for review August 13, 2026 17:30
Copilot AI balanced review requested due to automatic review settings August 13, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Makes threat-detect installation failures non-fatal in warn mode while preserving strict and expression-configured behavior.

Changes:

  • Applies detection continue-on-error policy to binary installation.
  • Adds default warn-mode coverage.
  • Regenerates 104 affected workflows.
Show a summary per file
File Description
pkg/workflow/threat_detection_steps.go Applies failure policy to installation.
pkg/workflow/threat_detection_isolation_test.go Tests default warn mode.
.github/workflows/docs-noob-tester.lock.yml Regenerates tolerant install step.
.github/workflows/ci-coach.lock.yml Regenerates tolerant install step.
.github/workflows/commit-changes-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/ab-testing-advisor.lock.yml Regenerates tolerant install step.
.github/workflows/agentic-token-trend-audit.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-session-insights.lock.yml Regenerates tolerant install step.
.github/workflows/breaking-change-checker.lock.yml Regenerates tolerant install step.
.github/workflows/daily-elixir-credo-snippet-audit.lock.yml Regenerates tolerant install step.
.github/workflows/daily-agentrx-trace-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cli-tools-tester.lock.yml Regenerates tolerant install step.
.github/workflows/blog-auditor.lock.yml Regenerates tolerant install step.
.github/workflows/cloclo.lock.yml Regenerates tolerant install step.
.github/workflows/q.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-prompt-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/issue-monster.lock.yml Regenerates tolerant install step.
.github/workflows/daily-astrostylelite-markdown-spellcheck.lock.yml Regenerates tolerant install step.
.github/workflows/daily-code-metrics.lock.yml Regenerates tolerant install step.
.github/workflows/daily-multi-device-docs-tester.lock.yml Regenerates tolerant install step.
.github/workflows/daily-choice-test.lock.yml Regenerates tolerant install step.
.github/workflows/duplicate-code-detector.lock.yml Regenerates tolerant install step.
.github/workflows/daily-evals-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-byok-ollama-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-file-diet.lock.yml Regenerates tolerant install step.
.github/workflows/changeset.lock.yml Regenerates tolerant install step.
.github/workflows/daily-formal-spec-verifier.lock.yml Regenerates tolerant install step.
.github/workflows/test-quality-sentinel.lock.yml Regenerates tolerant install step.
.github/workflows/daily-hippo-learn.lock.yml Regenerates tolerant install step.
.github/workflows/daily-assign-issue-to-user.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-merged-report.lock.yml Regenerates tolerant install step.
.github/workflows/deep-report.lock.yml Regenerates tolerant install step.
.github/workflows/impeccable-skills-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/pr-code-quality-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/cli-version-checker.lock.yml Regenerates tolerant install step.
.github/workflows/github-remote-mcp-auth-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-doc-healer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-experiment-report.lock.yml Regenerates tolerant install step.
.github/workflows/avenger.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-opt.lock.yml Regenerates tolerant install step.
.github/workflows/daily-regulatory.lock.yml Regenerates tolerant install step.
.github/workflows/craft.lock.yml Regenerates tolerant install step.
.github/workflows/daily-model-inventory.lock.yml Regenerates tolerant install step.
.github/workflows/daily-news.lock.yml Regenerates tolerant install step.
.github/workflows/daily-performance-summary.lock.yml Regenerates tolerant install step.
.github/workflows/daily-reliability-review.lock.yml Regenerates tolerant install step.
.github/workflows/daily-doc-updater.lock.yml Regenerates tolerant install step.
.github/workflows/agent-performance-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-token-consumption-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-mcp-concurrency-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/audit-workflows.lock.yml Regenerates tolerant install step.
.github/workflows/api-consumption-report.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cli-performance.lock.yml Regenerates tolerant install step.
.github/workflows/constraint-solving-potd.lock.yml Regenerates tolerant install step.
.github/workflows/daily-graft-intelligence.lock.yml Regenerates tolerant install step.
.github/workflows/chaos-pr-bundle-fuzzer.lock.yml Regenerates tolerant install step.
.github/workflows/ci-doctor.lock.yml Regenerates tolerant install step.
.github/workflows/contribution-check.lock.yml Regenerates tolerant install step.
.github/workflows/pr-sous-chef.lock.yml Regenerates tolerant install step.
.github/workflows/daily-compiler-quality.lock.yml Regenerates tolerant install step.
.github/workflows/design-decision-gate.lock.yml Regenerates tolerant install step.
.github/workflows/archivx-agentic-workflows-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-function-namer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-observability-report.lock.yml Regenerates tolerant install step.
.github/workflows/typist.lock.yml Regenerates tolerant install step.
.github/workflows/daily-ambient-context-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/prompt-clustering-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/code-scanning-fixer.lock.yml Regenerates tolerant install step.
.github/workflows/aw-failure-investigator.lock.yml Regenerates tolerant install step.
.github/workflows/architecture-guardian.lock.yml Regenerates tolerant install step.
.github/workflows/github-mcp-structural-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/daily-architecture-diagram.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-agent-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/daily-agent-of-the-day-blog-writer.lock.yml Regenerates tolerant install step.
.github/workflows/example-workflow-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-credit-limit-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-compiler-threat-spec-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/pr-description-caveman.lock.yml Regenerates tolerant install step.
.github/workflows/deployment-incident-monitor.lock.yml Regenerates tolerant install step.
.github/workflows/eslint-monster.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-cli-deep-research.lock.yml Regenerates tolerant install step.
.github/workflows/daily-awf-spec-compiler-surfacing.lock.yml Regenerates tolerant install step.
.github/workflows/ponytail-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-cache-strategy-analyzer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-max-ai-credits-test.lock.yml Regenerates tolerant install step.
.github/workflows/daily-caveman-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-model-resolution.lock.yml Regenerates tolerant install step.
.github/workflows/cli-consistency-checker.lock.yml Regenerates tolerant install step.
.github/workflows/detection-analysis-report.lock.yml Regenerates tolerant install step.
.github/workflows/agent-job-health.lock.yml Regenerates tolerant install step.
.github/workflows/daily-issues-report.lock.yml Regenerates tolerant install step.
.github/workflows/artifacts-summary.lock.yml Regenerates tolerant install step.
.github/workflows/agent-persona-explorer.lock.yml Regenerates tolerant install step.
.github/workflows/daily-geo-optimizer.lock.yml Regenerates tolerant install step.
.github/workflows/approach-validator.lock.yml Regenerates tolerant install step.
.github/workflows/mattpocock-skills-reviewer.lock.yml Regenerates tolerant install step.
.github/workflows/archie.lock.yml Regenerates tolerant install step.
.github/workflows/daily-testify-uber-super-expert.lock.yml Regenerates tolerant install step.
.github/workflows/daily-aw-cross-repo-compile-check.lock.yml Regenerates tolerant install step.
.github/workflows/daily-community-attribution.lock.yml Regenerates tolerant install step.
.github/workflows/daily-repo-chronicle.lock.yml Regenerates tolerant install step.
.github/workflows/daily-rendering-scripts-verifier.lock.yml Regenerates tolerant install step.
.github/workflows/copilot-pr-nlp-analysis.lock.yml Regenerates tolerant install step.
.github/workflows/auto-triage-issues.lock.yml Regenerates tolerant install step.
.github/workflows/claude-code-user-docs-review.lock.yml Regenerates tolerant install step.
.github/workflows/daily-fact.lock.yml Regenerates tolerant install step.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 106/106 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment on lines +227 to +228
if !strings.Contains(installStepBlock, "continue-on-error: true") {
t.Error("Install threat-detect binary step must set continue-on-error: true in warn mode")
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #52400

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #52400 does not have the 'implementation' label and has only 44 new lines of code in business logic directories (threshold: 100).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

This change weakens the strict-mode detection contract: the external detector install step now inherits the warn-mode continue-on-error behavior in every generated workflow, so a failed threat-detect download can be reported as success even when safe-outputs.threat-detection.continue-on-error: false is configured.

The blocking issue

The compiler already preserves strict vs warn behavior for the conclusion step, and buildInstallThreatDetectStep has tests/comments describing the same split. But the generated lockfiles in this PR all add continue-on-error: true unconditionally on the install step. That means a strict workflow cannot actually enforce binary availability anymore: the install step failure is swallowed before the detection job reaches the logic that is supposed to block safe outputs. Please add coverage for the strict and expression branches and make sure the recompiled workflows reflect the configured continue-on-error value instead of hard-coding true.

🔎 Code quality review by PR Code Quality Reviewer · gpt54 · 13.1 AIC · ⌖ 4.56 AIC · ⊞ 6.5K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ponytail review: one finding.

net: -8 lines possible.

Generated by ✂️ Ponytail Reviewer for #52400 · auto · 27.2 AIC · ⌖ 3.4 AIC · ⊞ 6.8K
Comment /ponytail to run again

// treat a failed/absent binary as a non-fatal detection failure via
// GH_AW_DETECTION_CONTINUE_ON_ERROR. Without continue-on-error on this install step, a
// transient download failure (e.g. a GitHub Releases CDN blip) would still mark this step —
// and therefore the whole detection job — as `failure`, even though the workflow logic

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L519-524: yagni: third copy of the continue-on-error resolution block (data.SafeOutputs.ThreatDetection lookup). Extract a resolveContinueOnError(data) (bool, string) helper used by all three buildStep functions.

@github-actions github-actions Bot mentioned this pull request Aug 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs — the root cause (CDN failure surfacing as job failure despite warn-mode tolerance) is precisely addressed.

📋 Summary

Positive Highlights

  • buildInstallThreatDetectStep correctly mirrors the continue-on-error resolution already used in buildDetectionConclusionStep and buildThreatDetectionAnalysisStep — consistent across the whole detection flow.
  • ✅ The IsContinueOnError() default (nil → true) means strict-mode workflows are not accidentally made more permissive.
  • ✅ 104 lock files recompiled consistently — no partial rollout risk.

Open Item

The existing review comment on line 228 of threat_detection_isolation_test.go notes that strict-mode (field omitted) and expression-mode branches are untested. That remains the only gap.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 38.1 AIC · ⌖ 6.56 AIC · ⊞ 7K
Comment /matt to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Quality Sentinel 🧪

PR #52400 — Make threat-detect binary install step continue-on-error in warn mode

Analysis Summary

Metric Result
Modified test files 1
Modified test functions 1 (TestExternalDetectorPath)
Lines added to tests 14
Lines added to production 30
Test inflation ratio 0.47:1 ✅
Build tags ✅ Present
Mock library violations None

Test Changes

File: pkg/workflow/threat_detection_isolation_test.go
Test: TestExternalDetectorPath

Modified test assertion details

The test adds a new behavioral contract verification (lines 216–229):

  1. Design invariant: Verifies that in warn mode (where workflows tolerate a missing threat-detect binary), the install step itself is marked continue-on-error: true to prevent transient download failures from cascading to job failure.

  2. Assertion strategy:

    • Locates the "Install threat-detect binary" step in the compiled detection job
    • Extracts the install step block boundaries
    • Asserts presence of continue-on-error: true within that block
  3. Value if deleted: High — this guards against a subtle failure mode where download transience could cause false negatives in threat detection despite the workflow logic gracefully handling missing binaries.

  4. Classification:

    • Design test: ✅ Verifies a contract guarantee
    • Edge-case coverage: ✅ Tests error-tolerance scenario (continue-on-error mode)
    • Behavioral contract: ✅ Enforces user-visible workflow resilience

Quality Score: 85/100 ✅ Excellent

Score breakdown:

  • Design contract tests: 100% (1/1 modified test is behavioral) → 40 pts
  • Edge-case coverage: Test includes error-tolerance scenario → 30 pts
  • Duplication: No duplicates → 20 pts
  • Test inflation: 0.47:1 ratio (well under 2:1 threshold) → 10 pts

Implementation test ratio: 0% (0/1 = 0%, threshold: 30%) ✅ Passes

Flags & Approvals

No violations detected:

  • Go build tags present on line 1
  • No Go mock library usage
  • Assertion-to-test ratio appropriate (1 added assertion is justified by the behavioral contract)
  • Test file changes proportionate to production changes

Recommendation

APPROVE — This is a focused, high-value test extension that adds genuine behavioral coverage for a resilience guarantee in warn mode. The addition validates that the compiled workflow correctly implements continue-on-error semantics for transient failures.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is correct — lock files are regenerated to include continue-on-error: true on the Install threat-detect binary step, matching the compiler logic already in threat_detection_steps.go (lines 537–540). All changed files are generated lock files; no source code modifications are required.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · sonnet46 · 34.2 AIC · ⌖ 6.34 AIC · ⊞ 5.4K

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Triage nudge from sous-chef. Please address the remaining review feedback, refresh the branch if needed, and run the pr-finisher skill before handing this PR back to maintainers.

Outstanding review items (newest first):

No failed checks were included in the compact candidate set.
Branch update was requested automatically for this run.
Run: https://github.com/github/gh-aw/actions/runs/31729392187

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 26 AIC · ⌖ 5.66 AIC · ⊞ 8.5K ·
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔍 PR Triage

Category: chore · Risk: medium · Priority score: 40/100 (impact 15, urgency 15, quality 10)
Recommended action: defer

Score breakdown:

  • Impact: Hardens CI resilience to CDN blips (continue-on-error for threat-detect installer) but touches 107 files (mostly regenerated lock.yml).
  • Urgency: Moderate — mitigates a low-frequency CI flake.
  • Quality: CHANGES_REQUESTED present; no check-run data returned. Needs author follow-up before merge.

⚠️ Outstanding CHANGES_REQUESTED review — please address before proceeding.

Generated by 🔧 PR Triage Agent · auto · 71.4 AIC · ⌖ 2.63 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: chore · Risk: medium · Priority: medium · Score: 50/100

Impact Urgency Quality
22/50 18/30 10/20

Recommended action: fast_track

Makes threat-detect/firewall binary installers continue-on-error in warn mode, hardening against transient CDN-blip failures across PR-gate workflows. CI status pending (no check-runs yet, commit status pending). Note: unusually high changed-file count (107) — worth a quick sanity check that scope matches the stated fix.

Generated by 🔧 PR Triage Agent · auto · 58.4 AIC · ⌖ 2.92 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

Triage: chore / risk: medium

Score: 65/100 (impact+urgency+quality) · Priority: high · Action: fast_track

Makes threat-detect installer continue-on-error to stop recurring PR-gate failures from CDN blips. Touches 107 workflow files (mechanical), CI pending.

Automated PR triage — see full report issue for details.

Generated by 🔧 PR Triage Agent · auto · 48.2 AIC · ⌖ 2.57 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage\n\n- Category: chore\n- Risk: low\n- Priority: low\n- Score: 31/100 (impact 10 + urgency 15 + quality 6)\n- Recommended action: batch_review\n- Batch: workflow-metadata-chores\n

Generated by 🔧 PR Triage Agent · auto · 62.8 AIC · ⌖ 2.76 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

  • Category: chore
  • Risk: low
  • Priority: medium (score: 41/100)
  • Score breakdown: impact 12/50, urgency 15/30, quality 14/20
  • Recommended action: batch_review
  • Batch: chore-workflow-updates

Automated triage by PR Triage Agent.

Generated by 🔧 PR Triage Agent · auto · 64.3 AIC · ⌖ 2.94 AIC · ⊞ 7.8K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw-failures] [P0] Harden AWF/threat-detect binary installers against transient GitHub Releases connection failures

4 participants