Compiler model inventory and AWF SUPPORTED_COPILOT_MODELS drift on every release; clean compile, guaranteed runtime abort

[crmitchelmore]

Versions: gh-aw v0.83.4 (compiler), bundled AWF v0.27.42 (constants.DefaultFirewallVersion), Copilot engine 1.0.70.

Summary

The compiler's model inventory and the firewall's SUPPORTED_COPILOT_MODELS allowlist are maintained independently and drift apart on every release. A model that gh-aw knows about compiles cleanly with 0 error(s), 0 warning(s) in strict mode and then hard-aborts host-side, before any container starts, on every run. There is no compile-time, lint-time or test-time signal — the first indication is a failed run in a downstream repo.

The two components release fast enough that this is a recurring maintenance tax rather than a one-off: 27 gh-aw-firewall releases and 21 gh-aw releases in the last 30 days. Because the firewall version is transitively pinned by the compiler version, every gh-aw bump can silently change which models are runnable, in either direction.

Current instance

gh-aw v0.83.4 added to pkg/cli/data/models.json under the github-copilot provider (changeset patch-model-inventory-2026-07-27.md):

• claude-opus-5
• gpt-5.6-sol, gpt-5.6-luna, gpt-5.6-terra

AWF v0.27.42 — the version v0.83.4 pins, and the newest firewall release — does not carry any of them in SUPPORTED_COPILOT_MODELS. That set is unchanged from v0.27.38 except for the addition of auto:

$ diff <(models @ v0.27.38) <(models @ v0.27.42)
1a2
>   'auto',

So this compiles with 0 errors:

engine:
  id: copilot
  model: claude-opus-5

and then every run of that workflow dies at preflight:

Error: model 'claude-opus-5' is unsupported or unrecognized by this AWF version.

validateCopilotModelOption() is called unconditionally from src/commands/validators/config-assembly.ts:76 during config assembly, so this is a total outage for the workflow, not a degraded run.

The drift is also internal to the firewall

The firewall's own docs/model-api-mapping.json at v0.27.42 already routes both families:

{ "family": "claude-opus-5", "patterns": ["claude-opus-5*"] }
{ "family": "gpt-5.6", "patterns": ["gpt-5.6*"],
  "notes": "Supports both endpoints. Includes gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna per OpenAI SDK ChatModel." }

So endpoint routing knows the models while the validator refuses them. ai-credits-pricing.js is missing them too, which is a second, separate gate (unknown_model_ai_credits) for any workflow setting max-ai-credits.

There is a CI guard for exactly one of the three pairs — src/copilot-model-catalog-sync.test.ts asserts pricing → allowlist only:

describe('SUPPORTED_COPILOT_MODELS ↔ ai-credits-pricing catalog sync')
  it('every Copilot CLI model in ai-credits-pricing.js appears in SUPPORTED_COPILOT_MODELS')

Nothing asserts model-api-mapping → allowlist, and nothing at all connects the compiler's inventory to the firewall's.

Why this is painful downstream

We pin a compiler version and distribute a catalogue of workflows to consumer repos. To pick a model safely today we have to:

1. read constants.DefaultFirewallVersion out of the pinned compiler,
2. fetch src/copilot-model.ts from that firewall tag and read the Set literal,
3. mirror it by hand into a unit test, and re-verify on every compiler bump,
4. separately check ai-credits-pricing.js for anything setting max-ai-credits.

That is the only reliable method we have found, because a clean strict-mode compile carries no information here. GH_AW_INFO_MODEL_COSTS does not help either — the compiler serialises its own pricing into the lock while the firewall's credit guard imports its own bundled catalogue.

Suggested remediation

Roughly in order of preference:

1. Validate engine.model at compile time against the allowlist of the firewall version being pinned, and fail (or warn in non-strict mode) when it is not a member. The compiler already knows which firewall it pins, so this is a closed-world check. This is the one that removes the class of bug.
2. Extend the existing sync guard to all three catalogues in gh-aw-firewall — SUPPORTED_COPILOT_MODELS ↔ ai-credits-pricing.js ↔ docs/model-api-mapping.json — so a model can't be half-added.
3. Add the allowlist to the compiler's release surface in a machine-readable form (e.g. emit it into the lock, or expose gh aw models --runnable), so consumers can assert against it without scraping a .ts file from another repo at a tag.
4. Failing all of the above, treat the allowlist as part of the compiler's release contract and update both in the same release, so that a model appearing in models.json is runnable by definition.

Related: #46306 covers a different path to the same outcome (provider-scoped engine.model accepted by the compiler, rejected by AWF at runtime). Both stem from the compiler validating against a broader catalogue than the firewall enforces.

[pelikhan]

@lpcox

[pelikhan]

/scout review plans for adding guards

[github-actions]

🏕️ Lost in the wilderness! Scout failed. Sending search party...

[crmitchelmore]

Filed the firewall-side counterpart as github/gh-aw-firewall#6682.

Root cause found while checking whether the models had landed there: docs/model-api-mapping.json has a scheduled updater ("chore: update model-to-API mapping", e.g. github/gh-aw-firewall#6622) whose PRs change exactly that one file, while SUPPORTED_COPILOT_MODELS and ai-credits-pricing.js are hand-edited. So the automated feed lands in the one catalogue that isn't covered by the existing sync guard, and the compiler's inventory tracks the feed rather than the gate.

That also makes this the third recurrence, after github/gh-aw-firewall#5969 (claude-fable-5) and #5991 (claude-sonnet-5) — both fixed per-model.

The two asks are complementary: #6682 proposes closing the triangle inside the firewall so the updater PR itself goes red; this issue asks for the compile-time check so gh-aw stops emitting locks its own bundled firewall will refuse.