api-proxy: fallback unavailable `gpt-5.x` requests to highest available family model by Copilot · Pull Request #3602 · github/gh-aw-firewall

Copilot · 2026-05-22T18:21:43Z

The proxy previously forwarded unknown model IDs unchanged when no alias matched, causing avoidable upstream 400s (e.g. gpt-5.5 on /chat/completions) and retry loops. This change adds in-proxy fallback selection for unavailable gpt-5.<minor> requests using the already-cached provider model list.

Model resolution behavior
- In resolveModel(), after alias and direct-match checks fail, the resolver now detects gpt-5.<minor> requests and selects the highest available gpt-5.* model for the current provider.
- Reuses existing version ordering (compareByVersion) to keep selection deterministic and consistent with alias candidate ranking.
- Emits a resolver log entry when fallback is applied:
  - requested model "gpt-5.5" not available, falling back to "gpt-5.4".
Body rewrite path
- rewriteModelInBody() now rewrites unavailable requests like gpt-5.5 to the selected fallback (gpt-5.4) instead of returning null and letting the invalid model pass through upstream.
Focused coverage
- Added resolver-level test for unavailable gpt-5 minor fallback.
- Added request-body rewrite test verifying rewritten model output for the same scenario.

// Before: null (proxy forwards "gpt-5.5" unchanged -> upstream 400)
// After: resolves to highest available gpt-5.* model
resolveModel('gpt-5.5', aliases, { copilot: ['gpt-5.2', 'gpt-5.4', 'gpt-4.1'] }, 'copilot')
// => { resolvedModel: 'gpt-5.4', ... }

lpcox · 2026-05-22T18:28:29Z

@copilot update the PR to include more comprehensive logging for model aliasing events. put them in token-diag.log and update the specs and schema to reflect their structure.

Copilot · 2026-05-22T18:33:22Z

@copilot update the PR to include more comprehensive logging for model aliasing events. put them in token-diag.log and update the specs and schema to reflect their structure.

Implemented in 81320f3. Model aliasing now emits structured diagnostics to token-diag.log (MODEL_ALIAS_RESOLUTION_STEP and MODEL_ALIAS_REWRITE), and I updated schema/spec coverage to validate the token-diag record structure (_schema, timestamp, event, optional data).

github-actions · 2026-05-22T18:35:21Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	95.98%	96.05%	📈 +0.07%
Statements	95.81%	95.87%	📈 +0.06%
Functions	98.02%	98.02%	➡️ +0.00%
Branches	89.44%	89.48%	📈 +0.04%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/config-writer.ts`	83.0% → 85.6% (+2.54%)	83.0% → 85.6% (+2.54%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-05-22T18:36:19Z

Smoke Test Results

PR #3596: Avoid gh /meta probe in firewall issue dispatcher

GitHub MCP: ✅ Retrieved PR Avoid gh /meta probe in firewall issue dispatcher #3596
GitHub.com HTTP: ❌ Pre-step data missing
File Write/Read: ❌ File not found at expected path

Status: FAIL

cc @Copilot @lpcox

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-05-22T18:36:25Z

Smoke Test Results: Claude Engine Validation

✅ GitHub API: 2 PR entries confirmed
✅ GitHub check: playwright_check PASS
✅ File verify: smoke-test-claude-26305292103.txt exists

Overall: PASS ✅

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-05-22T18:36:46Z

Smoke Test Results: PASS ✅

PR #3602: api-proxy: fallback unavailable gpt-5.x requests to highest available family model (by @Copilot, assigned to @lpcox @Copilot)

✅ GitHub MCP connectivity
❌ File write/read (file not found at expected path)
✅ BYOK inference (responding in offline mode)

Note: Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-05-22T18:37:30Z

API Proxy OTEL Tracing Validation Results

✅ Scenario 1: Module Loading

Status: PASS

otel.js exports all expected functions: startRequestSpan, setTokenAttributes, endSpan, endSpanError, shutdown, isEnabled
Module initializes successfully with isEnabled() returning true

✅ Scenario 2: Test Suite

Status: PASS (33/33 tests)

All OTEL module tests pass in containers/api-proxy/otel.test.js
Tests cover: span creation, token attributes, parent context propagation, OTLP export, file fallback, error handling

✅ Scenario 3: Env Var Forwarding

Status: PASS

src/services/api-proxy-service.ts forwards OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS to api-proxy container
OTEL header vars marked as one-shot credentials in agent-environment.ts (hidden from agent)

✅ Scenario 4: Token Tracker Integration

Status: PASS

proxy-request.js:503 calls otel.setTokenAttributes() in the onUsage callback
Token usage data (input/output/cache tokens) attached to spans before they close
onSpanEnd callback triggers otel.endSpan() with HTTP status code

✅ Scenario 5: OTEL Diagnostics

Status: EXPECTED (no export when OTLP_ENDPOINT unset)

OTEL is initialized but uses file fallback (/var/log/api-proxy/otel.jsonl) when endpoint is not configured
This is correct behavior — network export is opt-in

Overall Result: ✅ All scenarios PASS

The api-proxy OTEL integration is production-ready:

Spans created for each proxied LLM request with GenAI semantic conventions
Token usage attributes (gen_ai.usage.*) correctly attached
Parent context propagation from workflow trace works (via GITHUB_AW_OTEL_TRACE_ID/PARENT_SPAN_ID)
OTLP export routes through Squid proxy when configured
Graceful file fallback when OTLP endpoint is not set

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

127.0.0.1
api.example.com
api.openai.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "127.0.0.1"
    - "api.example.com"
    - "api.openai.com"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-05-22T18:37:42Z

Smoke Test Results

PR Stabilize test-coverage-reporter by isolating main-action unit tests from DinD probing #3595: Stabilize test-coverage-reporter... ✅
GitHub MCP: ❌ (mcpscripts not found)
GitHub.com Connectivity: ❌ (SSL error)
File Writing: ✅
Bash Tool: ✅

Overall Status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

Copilot

Pull request overview

This PR improves AWF’s API proxy model resolution by adding an in-proxy fallback for unavailable gpt-5.<minor> requests (e.g. gpt-5.5), selecting the highest available gpt-5.* model from the provider’s cached model list to avoid upstream 400s and retry loops. It also adds structured token diagnostics logging for model alias resolution/rewrite steps and corresponding test coverage.

Changes:

Extend resolveModel() / rewriteModelInBody() to fall back unavailable gpt-5.<minor> requests to the highest available gpt-5.* model for the provider.
Emit structured JSONL diagnostics to token-diag.log (when AWF_DEBUG_TOKENS=1) for model resolution steps and rewrites.
Add targeted unit tests for the new fallback behavior and for structured diagnostics logging/schema validation.

Show a summary per file

File	Description
containers/api-proxy/model-resolver.js	Adds `gpt-5.<minor>` unavailable fallback to highest available `gpt-5.*` model using existing version ordering.
containers/api-proxy/model-resolver.test.js	Adds resolver and body-rewrite tests covering the new `gpt-5.<minor>` fallback behavior.
containers/api-proxy/server.js	Writes structured diagnostics for model resolution/rewrite via `token-persistence.diag()`.
containers/api-proxy/server.models.test.js	Adds test verifying structured `token-diag.log` records are emitted for model alias rewrites.
containers/api-proxy/token-persistence.js	Introduces token-diag schema helpers (record builder + validator) and switches diag logging to structured JSONL records.
containers/api-proxy/token-tracker.schema.test.js	Adds schema-helper tests for `token-diag` record building and validation.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 6/6 changed files
Comments generated: 0

github-actions · 2026-05-22T18:38:09Z

Smoke: FAIL

PRs: Stabilize test-coverage-reporter by isolating main-action unit tests from DinD probing; Remove unused export from ProbeResult interface
✅ GitHub PR read
❌ safeinputs-gh unavailable
✅ Playwright title contains GitHub
❌ Tavily unavailable
✅ File/bash and npm build
❌ Discussion query tool unavailable; fallback comment posted
Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-05-22T18:38:14Z

Chroot Runtime Version Test Results

The chroot environment was tested to verify runtime versions match the host:

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌ NO
Node.js	v24.15.0	v22.22.3	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall Result: ❌ FAILED

The chroot environment has different Python and Node.js versions than the host. Only Go versions match. This indicates the bind mounts are correctly exposing host binaries, but the versions differ due to the base Ubuntu 22.04 image having older runtime versions than the GitHub Actions runner.

Tested by Smoke Chroot

github-actions · 2026-05-22T18:38:27Z

Smoke Test: Services Connectivity — ❌ FAIL

❌ Redis (host.docker.internal:6379) — timeout/no response
❌ PostgreSQL pg_isready (host.docker.internal:5432) — no response
❌ PostgreSQL query — not attempted (pg_isready failed)

Result: FAIL — AWF sandbox cannot reach GitHub Actions service containers

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-05-22T18:41:40Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

All build and test operations completed successfully across all 18 projects in 8 different programming language ecosystems.

Generated by Build Test Suite for issue #3602 · ● 15.7M · ◷

Initial plan

fb37804

Copilot AI assigned Copilot and lpcox May 22, 2026

Copilot started work on behalf of lpcox May 22, 2026 18:21 View session

Copilot AI linked an issue May 22, 2026 that may be closed by this pull request

feat(api-proxy): validate requested model against available models before forwarding #3601

Closed

feat(api-proxy): fallback unavailable gpt-5 minor models

298a566

Copilot AI changed the title ~~[WIP] Validate requested model against available models before forwarding~~ api-proxy: fallback unavailable gpt-5.x requests to highest available family model May 22, 2026

Copilot AI requested a review from lpcox May 22, 2026 18:26

Copilot finished work on behalf of lpcox May 22, 2026 18:26

Copilot started work on behalf of lpcox May 22, 2026 18:28 View session

feat(api-proxy): add structured model alias diagnostics to token-diag

81320f3

Copilot finished work on behalf of lpcox May 22, 2026 18:33

lpcox marked this pull request as ready for review May 22, 2026 18:34

Copilot AI review requested due to automatic review settings May 22, 2026 18:34

Copilot started reviewing on behalf of lpcox May 22, 2026 18:34 View session

github-actions Bot mentioned this pull request May 22, 2026

[aw] Smoke OpenCode failed #3603

Closed

github-actions Bot added smoke-claude smoke-copilot-byok labels May 22, 2026

Copilot AI reviewed May 22, 2026

View reviewed changes

github-actions Bot mentioned this pull request May 22, 2026

[aw] Security Guard failed #3604

Closed

github-actions Bot added the build-test label May 22, 2026

lpcox merged commit 7a64531 into main May 22, 2026
69 of 73 checks passed

lpcox deleted the copilot/feat-api-proxy-validate-models branch May 22, 2026 19:32

This was referenced May 22, 2026

feat(api-proxy): add middle-power model fallback with stale-cache recovery #3607

Merged

[aw] No-Op Runs #2911

Open

[aw] Smoke Codex failed #3627

Closed

Uh oh!

Conversation

Copilot AI commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lpcox commented May 22, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented May 22, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented May 22, 2026

Smoke Test Results: Claude Engine Validation

Uh oh!

github-actions Bot commented May 22, 2026

Smoke Test Results: PASS ✅

Uh oh!

github-actions Bot commented May 22, 2026

API Proxy OTEL Tracing Validation Results

✅ Scenario 1: Module Loading

✅ Scenario 2: Test Suite

✅ Scenario 3: Env Var Forwarding

✅ Scenario 4: Token Tracker Integration

✅ Scenario 5: OTEL Diagnostics

Uh oh!

github-actions Bot commented May 22, 2026

Smoke Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

github-actions Bot commented May 22, 2026

Smoke: FAIL

Uh oh!

github-actions Bot commented May 22, 2026

Chroot Runtime Version Test Results

Uh oh!

github-actions Bot commented May 22, 2026

Smoke Test: Services Connectivity — ❌ FAIL

Uh oh!

github-actions Bot commented May 22, 2026

🏗️ Build Test Suite Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 22, 2026 •

edited

Loading