Skip to content

Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot Sub Agents#42869

Closed
pelikhan with Copilot wants to merge 2 commits into
mainfrom
copilot/aw-failures-fix-smoke-copilot-sub-agents
Closed

Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot Sub Agents#42869
pelikhan with Copilot wants to merge 2 commits into
mainfrom
copilot/aw-failures-fix-smoke-copilot-sub-agents

Conversation

Copilot AI commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot Sub Agents was hard-failing on every run before reaching the model: the Copilot SDK returned No model available. Check policy enablement, but the harness did not classify it as a model/policy failure and burned all retries. This change teaches the harness to recognize that error as non-retryable and repins the smoke workflow to a Copilot model that is already serving successfully.

  • Error classification

    • Extend Copilot model-unavailable detection to match the SDK policy-enablement error:
      • No model available. Check policy enablement under GitHub Settings > Copilot
    • Route that error through the existing model_not_supported_error path so the workflow reports a clear configuration failure instead of partial_execution.
  • Retry behavior

    • Stop retrying deterministic Copilot SDK model/policy failures in the harness.
    • Preserve existing retry behavior for transient partial-execution paths.
  • Smoke workflow repin

    • Change the parent model in smoke-copilot-sub-agents from gpt-5.3-codex to gpt-5.4.
    • Regenerate the compiled lock file so the runtime workflow matches the source workflow.
  • Regression coverage

    • Add focused tests for:
      • direct detection of the Copilot SDK no-model/policy-enablement error
      • non-retry behavior for that error
      • workflow-output classification via detectCopilotErrors(...)
const MODEL_NOT_SUPPORTED_PATTERN =
  /(?:The requested model is not supported|No model available\.\s*Check policy enablement under GitHub Settings\s*>\s*Copilot)/i;

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix smoke Copilot Sub Agents for model accessibility Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot Sub Agents Jul 2, 2026
Copilot AI requested a review from pelikhan July 2, 2026 01:37
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Great work on this one — the No model available. Check policy enablement error was a real pain point in the smoke harness, and this PR handles it cleanly end-to-end.

The change is well-scoped: the regex extension in detect_agent_errors.cjs targets exactly the new SDK error string, the non-retry routing in copilot_harness.cjs is surgical, the smoke workflow repin is the logical follow-through, and the regression tests in both .test.cjs files give solid coverage of the new detection path and the non-retry behavior.

This looks ready for review. 🚀

Generated by ✅ Contribution Check · 302.1 AIC · ⌖ 26.3 AIC · ⊞ 6.3K ·

@pelikhan pelikhan closed this Jul 2, 2026
@github-actions github-actions Bot deleted the copilot/aw-failures-fix-smoke-copilot-sub-agents branch July 9, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants