Honor detection continue-on-error when external detector result file is missing#40790
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the external threat-detect conclude step generation so missing detection_result.json can be handled without incorrectly failing the detection job when continue-on-error mode is enabled, and extends the isolation test to assert the new emitted YAML content.
Changes:
- Add a missing-result preflight in
buildExternalDetectorConcludeStepthat emits fallback step outputs/env and exits successfully in continue-on-error mode. - Keep strict-mode behavior as a hard failure when the result file is missing.
- Update
TestExternalDetectorPathto assert the generated YAML contains the missing-result fallback markers.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/threat_detection.go | Adds shell-level missing-result handling and emits fallback outputs/env before running threat-detect conclude. |
| pkg/workflow/threat_detection_isolation_test.go | Adds assertions that the external-detector YAML includes the new fallback content. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 3
| " echo \"conclusion=unknown\" >> \"$GITHUB_OUTPUT\"\n", | ||
| " echo \"success=false\" >> \"$GITHUB_OUTPUT\"\n", | ||
| " echo \"reason=agent_failure\" >> \"$GITHUB_OUTPUT\"\n", | ||
| " echo \"GH_AW_DETECTION_CONCLUSION=unknown\" >> \"$GITHUB_ENV\"\n", | ||
| " echo \"GH_AW_DETECTION_REASON=agent_failure\" >> \"$GITHUB_ENV\"\n", |
| if !strings.Contains(detectionSection, "conclusion=unknown") { | ||
| t.Error("External detector path must degrade to conclusion=unknown when result file is missing in continue-on-error mode") | ||
| } |
| if !strings.Contains(detectionSection, "GH_AW_DETECTION_CONCLUSION=unknown") { | ||
| t.Error("External detector path must export GH_AW_DETECTION_CONCLUSION=unknown for missing result fallback") | ||
| } |
| coeEnvLine, | ||
| " run: |\n", | ||
| fmt.Sprintf(" threat-detect conclude --result-file %s\n", shellEscapeArg(constants.ThreatDetectionResultPath)), | ||
| fmt.Sprintf(" result_file=%s\n", shellEscapeArg(constants.ThreatDetectionResultPath)), |
There was a problem hiding this comment.
@copilot refactor to a script file in actions/setup/sh + add tests
There was a problem hiding this comment.
Done in 5434a35. I refactored the external detection conclude logic into actions/setup/sh/conclude_threat_detection.sh, switched workflow generation to call that script, and added tests in pkg/workflow/threat_detection_conclude_script_test.go plus updated isolation assertions.
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot run pr-finisher skill |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot run pr-finisher skill |
Ran a pr-finisher pass on this branch. Local validation is green ( |
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ Caution agentic threat detected DetailsThe threat detection engine failed to produce results. Review the workflow run logs for details. |
|
📰 BREAKING: Smoke Copilot - AOAI (Entra) is now investigating this pull request. Sources say the story is developing... |
Agent Container Tool Check
Result: 12/12 tools available ✅ Overall Status: PASS
|
|
Caution agentic threat detected DetailsThe threat detection engine failed to produce results. Review the workflow run logs for details. Smoke Test Results
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
|
Caution agentic threat detected DetailsThe threat detection engine failed to produce results. Review the workflow run logs for details. Smoke test FAIL
Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
Caution agentic threat detected DetailsThe threat detection engine failed to produce results. Review the workflow run logs for details. Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment. Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
Smoke Test Results: Overall: PASS
|
|
Caution agentic threat detected DetailsPotential security threats were detected in the agent output. Review the workflow run logs for details. Honor detection continue-on-error when external detector result file is missing ❌ github tool FAIL Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
|
Caution agentic threat detected DetailsPotential security threats were detected in the agent output. Review the workflow run logs for details. Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment. Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.
Details
Potential security threats were detected in the agent output.
Review the workflow run logs for details.
Smoke cave note: review tool path worked.
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
📰 BREAKING: Report filed by Smoke Copilot · 429 AIC · ⌖ 13.7 AIC · ⊞ 19.2K
|
Caution agentic threat detected DetailsThe threat detection engine failed to produce results. Review the workflow run logs for details. Smoke Test: Claude — Run 27980364818Core #1-12: all passed (MCP, GH CLI, Build, Playwright, Tavily, Files, Bash, Discussion, AW status, Slack, Code Scan, Check Run) ✅ PR Review #13-19: Update PR ✅ · Review Comments ✅ · Submit Review ✅ · Resolve Thread ✅ · Add Reviewer ✅ · Push Branch ✅ · Close PR Overall: PARTIAL (only #19 skipped — no safe test PR) Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
|
There was a problem hiding this comment.
Caution
agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.
Details
The threat detection engine failed to produce results.
Review the workflow run logs for details.
💥 Automated smoke test review - all systems nominal!
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "safebrowsingohttpgateway.googleapis.com"
- "www.google.com"See Network Configuration for more information.
💥 [THE END] — Illustrated by Smoke Claude · 114.8 AIC · ⊞ 8.5K
| coeEnvLine, | ||
| " run: |\n", | ||
| fmt.Sprintf(" threat-detect conclude --result-file %s\n", shellEscapeArg(constants.ThreatDetectionResultPath)), | ||
| fmt.Sprintf(" bash \"${RUNNER_TEMP}/gh-aw/actions/conclude_threat_detection.sh\" %s\n", shellEscapeArg(constants.ThreatDetectionResultPath)), |
There was a problem hiding this comment.
Nice — replacing the inline threat-detect conclude with a sourced shell script keeps the conclude logic testable. Worth a comment noting the script path contract.
| @@ -1279,7 +1279,7 @@ func (c *Compiler) buildExternalDetectorConcludeStep(data *WorkflowData) []strin | |||
| " DETECTION_AGENTIC_EXECUTION_OUTCOME: ${{ steps.detection_agentic_execution.outcome }}\n", | |||
There was a problem hiding this comment.
Consider asserting that the RUNNER_TEMP-based script exists before invoking, so a missing detector script fails clearly rather than via a bash not-found error.
|
@copilot review failures in detection job. https://github.com/github/gh-aw/actions/runs/27980364818 |
Reviewed run 27980364818. The workflow and detection job both completed successfully, but the detection execution step logged an internal detector failure: |
|
Please summarize any remaining blockers around the continue-on-error handling.
|
|
@copilot investigate why Claude was not found by the detection engine. Check differences with agent job. https://github.com/github/gh-aw/actions/runs/27980364818/job/82810895179#step:18:1 |
pelikhan
deleted the
copilot/aw-failures-honor-detection-continue-on-error
branch
A transient failure in the external detection agent can leave
detection_result.jsonabsent; previously, detection conclude handling could hard-fail the detection job even whenGH_AW_DETECTION_CONTINUE_ON_ERROR=true. This change keeps missing-result handling mode-aware so detection failures no longer incorrectly dominate otherwise successful runs.External detector conclude path: script-backed missing-result fallback
actions/setup/sh/conclude_threat_detection.sh, invoked frombuildExternalDetectorConcludeStep.RUN_DETECTION == trueGH_AW_DETECTION_CONTINUE_ON_ERRORconclusion=warningsuccess=falsereason=agent_failureGH_AW_DETECTION_CONCLUSION=warningandGH_AW_DETECTION_REASON=agent_failureexit 1).Contract preservation
id: detection_conclusionand preserves downstream output/env contracts.Coverage update
threat_detection_isolation_test.gonow asserts the external-detector YAML invokesconclude_threat_detection.shand preserves required conclude env wiring.threat_detection_conclude_script_test.goto validate:threat-detect concludewhen result file exists.Caution
agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.
Details
The threat detection engine failed to produce results.
Review the workflow run logs for details.
\n\n---\n✨ PR Review Safe Output Test - Run 27980364818
Warning
Firewall blocked 6 domains
The following domains were blocked by the firewall during workflow execution:
accounts.google.comandroid.clients.google.comclients2.google.comcontentautofill.googleapis.comsafebrowsingohttpgateway.googleapis.comwww.google.comSee Network Configuration for more information.