🤖 feat: GPT-5.6 explicit prompt cache breakpoints for direct OpenAI

[ThomasK33]

Summary

Adds hybrid prompt caching for official, direct-OpenAI GPT-5.6-family requests: one explicit prompt_cache_breakpoint at the end of Mux's stable system/developer instructions, plus the project-scoped prompt_cache_key on eligible Chat Completions requests. Request-wide caching stays implicit (no promptCacheOptions), so OpenAI keeps placing its automatic latest-message breakpoint for growing conversations and tool loops.

Background

GPT-5.6 (via @ai-sdk/openai 4.0.11) supports explicit content-level cache breakpoints. Mux's system prefix is stable across turns and context resets within a workspace, so pinning one explicit breakpoint at that boundary lets a fresh context (e.g. after Reset Context, Preserve History) reuse the cached instruction prefix instead of re-writing it — while implicit mode continues to cover the conversation tail.

Implementation

• openaiExplicitPromptCachingAvailable (cacheStrategy.ts): fail-closed eligibility gate — request namespace and resolved alias target must both be openai: GPT-5.6-family, route provider must be exactly openai, Codex OAuth precedence fails closed (wouldRouteOpenAIThroughCodexOauth), and any configured base URL must be the official https://api.openai.com endpoint (root or /v1). Gateways, custom endpoints, and missing route metadata all fail closed.
• createOpenAICachedSystemMessage: returns a structured SystemModelMessage with message-level providerOptions.openai.promptCacheBreakpoint: { mode: "explicit" }; the SDK serializes it to an input_text (Responses) / text (Chat Completions) block.
• streamManager.buildStreamRequestConfig now takes the resolved route: initialMetadata.routeProvider for primary requests and initialMetadataPatch.routeProvider for refusal-fallback rebuilds, so cache metadata cannot leak across routes in either direction. StreamRequestConfig.system widened to string | SystemModelMessage; the Anthropic cached-system path is untouched.
• Chat Completions promptCacheKey (providerOptions.ts): added only behind the stricter gate. The legacy Responses key spread (including its broader route-absent/passthrough behavior) is byte-for-byte unchanged.

Validation

• Wire-level tests (createOpenAI + capture fetch + streamText over synthetic SSE) prove breakpoint + cache-key serialization on both wire formats, absence of prompt_cache_options, and that cached_tokens/cache_write_tokens surface as inputTokenDetails.cacheReadTokens/cacheWriteTokens through the streaming parser.
• Live dogfooding on gpt-5.6-luna (official API key, isolated desktop sandbox):
  • Responses: first request wrote 14,445 cache tokens; after Reset Context, Preserve History, the raw request proved all isolation invariants (no pre-reset turns, no previous_response_id, unchanged prompt_cache_key, byte-identical system SHA-256) and read 14,410 cached tokens — explicit stable-prefix reuse. A follow-up read 14,429 (implicit latest-message reuse retained). Cost tab shows distinct cache-read/create rows priced at 0.1×/1.25×.
  • Chat Completions: breakpoint (text block) + key serialize on the live wire; write 10,145 and same-conversation read 10,082 confirmed. Cross-conversation explicit-prefix reads stayed 0 over ~8 min of retries despite byte-identical prefixes — an OpenAI-side CC behavior, not a request-shape defect (Responses, Mux's default, shows full reuse).
  • A first attempt silently routed via mux-gateway and the gate correctly failed closed (no breakpoint emitted) — live proof of the fail-closed design.
• Live constraint noted: GPT-5.6 Chat Completions rejects function tools with reasoning_effort != none, so the CC flow was validated with thinking off. Pre-existing behavior, unrelated to this change.

Risks

• Low, tightly scoped: the only behavioral changes for existing routes are (a) direct-OpenAI GPT-5.6 system prompts become structured messages with a breakpoint, and (b) direct-OpenAI GPT-5.6 Chat Completions gain prompt_cache_key. All other models, routes (gateways, Codex OAuth, custom endpoints), and Anthropic cache behavior are unchanged and covered by fail-closed tests.
• Refusal-fallback rebuilds re-evaluate eligibility from the fallback route patch, tested in both eligible→ineligible and ineligible→eligible directions.

---

📋 Implementation Plan

GPT-5.6 explicit prompt caching

Goal

Add GPT-5.6 explicit stable-prefix cache breakpoints while retaining OpenAI's implicit latest-message caching, then verify request shape, token accounting, and real cache reuse without changing non-GPT-5.6 routes.

Current state

• Branch is rebased onto origin/main at 1d6e0caa5.
• @ai-sdk/openai is now 4.0.11, which supports promptCacheOptions, content-part promptCacheBreakpoint, and cache_write_tokens parsing.
• Mux already supplies a stable project-scoped promptCacheKey for OpenAI Responses requests and already normalizes cache reads/writes through the usage pipeline.
• Anthropic has separate explicit cache-control transforms; the OpenAI implementation should share only provider-neutral mechanics where that reduces duplication, not Anthropic wire semantics.
• Direct production OpenAI routing resolves to routeProvider: "openai"; missing route metadata is a legacy/test-stub case and should fail closed for the new fields.
• StreamManager already receives the safe ProviderService.getConfig() map, including effective base-URL and OAuth-source metadata, and AIService already rebuilds fallback system/provider options from the fallback model and route.

Recommended approach

Use hybrid caching for eligible official, direct OpenAI GPT-5.6-family requests:

1. Keep request-wide mode implicit by omitting promptCacheOptions; GPT-5.6 already defaults to implicit mode and a 30-minute minimum TTL.
2. Add exactly one explicit breakpoint at the end of Mux's stable system/developer instructions.
3. Preserve OpenAI's automatic latest-message breakpoint so growing conversations and tool loops continue to reuse their prior prefixes.
4. Continue the existing project-scoped promptCacheKey behavior for Responses, and add the same key to eligible Chat Completions requests.
5. Fail closed for Codex OAuth, gateways, custom OpenAI-compatible providers, and non-official OpenAI base URLs until those routes are verified independently.

Estimated net product-code change: approximately +65 to +95 LoC. Test-only changes are excluded.

Scope and non-goals

In scope

• The GPT-5.6 base ID plus Sol, Terra, and Luna as canonical direct-OpenAI model strings (openai:gpt-5.6... at the implementation call sites), including supported date-suffixed IDs and openai: aliases mapped to those targets. Raw unprefixed strings are normalized before these call sites; the new capability helper itself does not infer a provider and fails closed when the model origin is missing or non-OpenAI.
• Both OpenAI Responses and Chat Completions wire formats when the resolved route is the official OpenAI API using API-key authentication.
• Main streams and refusal fallbacks rebuilt by StreamManager.
• Existing cache-read/write accounting, cost display, DevTools telemetry, and analytics ingestion.

Out of scope

• promptCacheOptions.mode: "explicit"; explicit-only mode would disable the useful automatic latest-message breakpoint.
• Explicit breakpoints on conversation turns, tool calls, tool results, files, or individual tools.
• Cache-key format changes, per-agent sharding, or new cache settings in the UI.
• Enabling the feature through Codex OAuth, Mux Gateway, OpenRouter, GitHub Copilot, Azure/OpenAI-compatible endpoints, or a configured custom OpenAI base URL.
• Sanitizing hand-authored providerExtras.promptCacheKey values; that generic expert escape hatch is pre-existing behavior, while this plan governs only fields generated by Mux.
• Changing the legacy internal providerMetadata.anthropic.cacheCreationInputTokens persistence key; it already carries provider-neutral AI SDK cache-write usage through the current pricing pipeline despite the historical name.

Implementation plan

Implement each phase test-first: add the failing behavioral assertion, make the smallest production change that satisfies it, then run the phase gate before continuing.

Phase 1 — Add a route-aware GPT-5.6 cache capability

1. In src/common/types/thinking.ts, export the existing isGpt56FamilyModel matcher instead of duplicating its model-ID regex.

2. In src/common/utils/ai/cacheStrategy.ts, add one pure openaiExplicitPromptCachingAvailable(modelString, routeProvider, providersConfig: ProvidersConfigMap) helper that:

   • accepts the canonical model-string shapes already supplied by AIService (openai:<model-id> for direct built-ins and openai: aliases); require the request model's parsed origin to be exactly openai, so raw unprefixed inputs and non-OpenAI namespaces fail closed instead of relying on default-provider inference;
   • resolves mappedToModel aliases through resolveModelForMetadata, independently requires the resolved capability target's origin to be openai, and only then applies the GPT-5.6 family predicate to the target model ID;
   • requires the backend-resolved route provider to be exactly openai; missing, legacy, gateway, or unknown route metadata fails closed even though older provider-option paths are more permissive;
   • reuses wouldRouteOpenAIThroughCodexOauth for API-key/OAuth precedence instead of duplicating provider-factory auth logic;
   • reads the safe openai.baseUrl / baseUrlResolved view already returned by ProviderService.getConfig(): absence of both means the SDK's official default, otherwise resolve the active value with baseUrl precedence and require HTTPS host api.openai.com, no credentials/query/hash/non-default port, and only the root or /v1 path with an optional trailing slash;
   • rejects malformed or non-official endpoint values and otherwise fails closed whenever eligibility cannot be established. Transport-level HTTP proxy environment variables are not endpoint overrides and stay outside this check.

3. Add createOpenAICachedSystemMessage, returning the exact AI SDK 7 input shape—message-level provider options with string content, not a content-part array:

   const cachedSystem = {
     role: "system",
     content: systemContent,
     providerOptions: {
       openai: {
         promptCacheBreakpoint: { mode: "explicit" },
       },
     },
   } satisfies SystemModelMessage;

   Return null for an empty system prompt or an ineligible route. Keep Anthropic cache-control helpers and wire semantics unchanged.

Quality gate: extend src/common/utils/ai/cacheStrategy.test.ts to cover the actual production shapes (openai:gpt-5.6... and openai: mapped aliases), every GPT-5.6 tier, rejection of raw unprefixed and non-OpenAI-origin strings, aliases whose resolved target is not OpenAI GPT-5.6, older models, missing/unknown/gateway routes, Codex OAuth precedence, no URL override, official explicit and baseUrlResolved URLs, custom/malformed/non-HTTPS URLs, empty system text, and the exact typed system-message shape.

Phase 2 — Attach the breakpoint at the system-instructions boundary

1. In src/node/services/streamManager.ts, widen only StreamRequestConfig.system to string | SystemModelMessage | undefined; keep PreparedModelFallback.system and public/startup inputs as strings.
2. Add an explicit resolved-route argument to buildStreamRequestConfig and let the builder use its existing getProvidersConfig() callback for endpoint/auth eligibility; do not add another provider-resolution path.
3. For the primary request, pass the already-resolved initialMetadata?.routeProvider from createStreamAtomically.
4. Preserve the existing AIService fallback rebuild: it already recreates the fallback system, messages, tools, headers, and provider options from next.canonicalModelString / next.routeProvider. Before streamInfo.initialMetadata is patched, pass prepared.data.initialMetadataPatch?.routeProvider directly into the fallback buildStreamRequestConfig call so the new system transform evaluates the fallback route rather than stale source metadata.
5. Inside buildStreamRequestConfig:
   • preserve the current Anthropic path, which moves a cached system message into messages and clears system;
   • otherwise call createOpenAICachedSystemMessage with the model, safe provider config, and resolved route;
   • when eligible, keep messages unchanged and replace the system string with the structured SystemModelMessage accepted by AI SDK 7;
   • do not add request-level promptCacheOptions, because omitted mode and TTL preserve hybrid implicit + explicit caching with the provider default lifetime.
6. Leave the existing unconditional allowSystemInMessages: true behavior unchanged. OpenAI's structured instructions remain in the system argument and do not require a new compatibility switch.

Quality gate: extend src/node/services/streamManager.test.ts to prove direct GPT-5.6 produces structured cached instructions, ineligible routes/endpoints preserve the original string, Anthropic behavior remains unchanged, and the fallback route patch controls the transformed fallback system. Extend src/node/services/aiService.test.ts only at the existing fallback seam to prove fallback provider options and route metadata are freshly built from the fallback model/route, including eligible→ineligible and ineligible→eligible Chat Completions cases.

Phase 3 — Supply the routing key on GPT-5.6 Chat Completions

1. In src/common/utils/ai/providerOptions.ts, reuse the same exact-route/auth/endpoint eligibility helper for the new Chat Completions behavior.
2. Keep the existing Responses promptCacheKey spread and scope exactly where it is. Its current route-absent, passthrough-gateway, Codex OAuth, and custom-endpoint behavior is legacy behavior and must not be moved behind the stricter helper in this patch.
3. For Chat Completions only, add the existing project/workspace-derived promptCacheKey when the request is eligible for direct official GPT-5.6 explicit caching. Continue omitting Responses-only fields such as truncation, include, and reasoning summaries from Chat Completions.
4. Do not emit promptCacheOptions; the provider defaults are the intended policy.

Quality gate: extend src/common/utils/ai/providerOptions.test.ts so direct GPT-5.6 Chat Completions with routeProvider: "openai" receives the cache key, while older models, mapped non-GPT-5.6 aliases, missing/unknown routes, gateways, Codex OAuth, and custom endpoints do not. Retain existing Responses assertions and add a GPT-5.6 Responses regression proving the legacy key remains unchanged for route-absent and passthrough cases.

Phase 4 — Prove SDK serialization and accounting behavior

1. Add deterministic production-path tests using createOpenAI, a non-networked capture fetch, and streamText for both provider formats, following the existing serialization pattern in src/common/utils/ai/providerOptions.test.ts:
   • Responses must serialize the structured system instructions to an input_text block with prompt_cache_breakpoint;
   • Chat Completions must serialize them to a text block with the same breakpoint;
   • both eligible formats must include prompt_cache_key;
   • neither format should contain prompt_cache_options or disable implicit mode;
   • set maxRetries: 0, return minimal valid SSE, fully consume each stream, and await the public usage result so the test exercises the same streaming parser used by Mux rather than relying on generateText parser sharing.
2. Put synthetic cached_tokens and cache_write_tokens in the final usage-bearing Chat Completions chunk and in the Responses response.completed.response.usage event, then assert AI SDK 4.0.11 exposes both as inputTokenDetails.cacheReadTokens and cacheWriteTokens for each wire format.
3. Extend src/common/utils/tokens/displayUsage.test.ts with a GPT-5.6 case proving Mux separates ordinary input, cache reads, and cache writes and applies the configured 0.1× read and 1.25× write prices. Do not rewrite the already-covered normalization pipeline.

Quality gate: run the focused suites before broader checks:

bun test \
  src/common/utils/ai/cacheStrategy.test.ts \
  src/common/utils/ai/providerOptions.test.ts \
  src/node/services/streamManager.test.ts \
  src/node/services/aiService.test.ts \
  src/common/utils/tokens/displayUsage.test.ts
make typecheck

Phase 5 — Full validation

1. Run formatting, lint, typechecking, and the relevant test suite through the repository targets:

   MUX_ESLINT_CONCURRENCY=1 make static-check

2. Re-run the focused tests in a fresh Bun process if the broad check exposes QuickJS/resource instability unrelated to these files.

3. Inspect the final diff and confirm no provider config schema, UI, analytics schema, or Anthropic cache behavior changed.

Dogfooding and evidence

Use a live official OpenAI API-key route because custom base URLs are intentionally excluded and a mock endpoint would not exercise the production eligibility gate. Use gpt-5.6-luna to minimize cost.

Setup

1. Confirm an OpenAI API key with GPT-5.6 access is available, Codex OAuth is not the selected auth path, and the OpenAI provider has no custom base URL.

2. Start an isolated desktop with provider settings copied into a temporary root:

   MUX_E2E=1 KEEP_SANDBOX=1 ELECTRON_DEBUG_PORT=9223 make dev-desktop-sandbox

3. Connect agent-browser to the Electron CDP port, enable Settings → General → API Debug Logs, and open the Right Sidebar DevTools and Cost tabs.

4. Use one workspace so its workspace environment block and full system prompt remain byte-identical. Save a sanitized copy of the first raw request and a SHA-256 of its system/developer text. After the first request, use the command palette action Reset Context, Preserve History to create a durable context boundary without changing the workspace or cache key. The post-reset raw request—not the cache counter alone—must prove that the prior conversation was sliced out before treating the next read as explicit stable-prefix reuse. Run requests sequentially and wait for each stream-end signal before continuing.

Responses flow

1. Start a recording before model selection and preserve it through the DevTools/Cost inspection:

   mkdir -p /tmp/mux-gpt56-cache/{screenshots,videos}
   agent-browser --session gpt56-cache connect 9223
   agent-browser --session gpt56-cache record start /tmp/mux-gpt56-cache/videos/responses-cache.webm

2. In workspace A, select gpt-5.6-luna on the Responses wire format and send a short-output prompt. Wait for the explicit stream-end signal, not a fixed sleep.

3. Capture the DevTools request showing:

   • a stable prompt_cache_key;
   • one system/developer content block containing prompt_cache_breakpoint: { mode: "explicit" };
   • no prompt_cache_options.mode: "explicit".

4. Capture the first response's nonzero Cache write value.

5. Invoke Reset Context, Preserve History in the same workspace, then send a different first user message. Capture the sanitized post-reset raw request and verify all four isolation invariants before interpreting the usage result: no pre-reset user/assistant/tool messages remain in input; no previous_response_id wire field (or camel-case previousResponseId metadata field) is present; prompt_cache_key is unchanged; and the system/developer block plus its prompt_cache_breakpoint is byte-identical or has the same SHA-256 as the first request. Then capture a nonzero Cache read value; with those invariants established, the hit demonstrates stable instruction-prefix reuse rather than implicit reuse of the prior conversation.

6. Send one follow-up in the new context and capture the longer cached prefix expected from the retained implicit latest-message breakpoint. Treat latency as supporting evidence only; token counters plus the raw-request isolation checks are the acceptance signal.

7. Capture the Cost tab showing distinct uncached input, cache-read, and cache-write token/cost rows, then stop the recording.

Chat Completions flow

1. Switch the direct OpenAI provider to Chat Completions and repeat the first request → context reset → unrelated request → follow-up sequence at low request rate.
2. Capture a request screenshot proving both prompt_cache_key and the system content breakpoint are serialized in the Chat Completions body.
3. For the post-reset Chat Completions request, repeat the same isolation proof against messages: no pre-reset turns, no previous-response identifier in raw request or metadata, unchanged cache key, and byte/hash-identical developer/system content.
4. Capture at least one cache write and one later cache read, then record the full flow to /tmp/mux-gpt56-cache/videos/chat-completions-cache.webm.

Evidence requirements

• Annotated screenshots for the Responses request marker, initial cache write, same-workspace post-reset cache read, Cost-tab breakdown, and Chat Completions request marker/read.
• Sanitized pre/post-reset request extracts for each wire format (or screenshots of those extracts) showing the active message list, absence of previous-response identifiers, unchanged cache key, and matching system/developer SHA-256 values.
• One watchable .webm per wire format, with pauses around request inspection and token results.
• Inspect every artifact for API keys, authorization headers, private repository content, or other secrets before attaching it with attach_file.
• If a live key/model is unavailable, report dogfooding as blocked rather than substituting a custom endpoint that bypasses the production gate.

Acceptance criteria

• Official direct OpenAI GPT-5.6 Responses and Chat Completions requests serialize one explicit breakpoint at the stable system/developer boundary and carry the stable cache key.
• Request-wide caching remains implicit; no code emits explicit-only mode or redundant TTL configuration.
• After a same-workspace durable context reset, the raw request proves that pre-reset turns and previous-response identifiers are absent while the cache key and system/developer bytes remain unchanged; only then does a nonzero cache read establish stable-prefix reuse, and a follow-up turn retains growing-conversation reuse.
• Older OpenAI models, gateways, Codex OAuth, custom providers/base URLs, missing/unknown routes, and non-OpenAI models receive no new Mux-generated message breakpoint or Chat Completions cache key.
• Existing Responses promptCacheKey behavior remains byte-for-byte unchanged, including its broader legacy route behavior.
• Mapped aliases inherit eligibility only when both the request namespace and resolved capability target are OpenAI, the target is GPT-5.6-family, and the resolved route/auth/base URL identify the official direct OpenAI API.
• Refusal fallback rebuilds reevaluate capability and route, preventing provider-specific cache metadata from leaking across models.
• Cache reads and writes reach DevTools, cost display, persisted usage, and analytics with GPT-5.6 pricing; no migration or provider-specific accounting rewrite is required.
• Focused tests, make typecheck, and MUX_ESLINT_CONCURRENCY=1 make static-check pass.
• Desktop dogfooding produces sanitized screenshots and video for both wire formats, or records the exact external credential/access blocker.

Risks and mitigations

• Unsupported endpoint fields: fail closed for OAuth, gateways, custom providers, and custom base URLs.
• SDK serialization drift: the capture-fetch wire test verifies both API formats rather than only testing intermediate TypeScript objects.
• Cache writes without reuse: use one stable breakpoint, preserve implicit mode, and verify a same-workspace post-boundary read before declaring success.
• Prefixes below 1,024 tokens: dogfood with Mux's normal full instruction prefix and confirm actual cache-write tokens; do not infer success from request shape alone.
• Hot project cache keys: keep the existing key format in this patch and avoid concurrent dogfood traffic. If production telemetry later shows hot keys, handle deterministic sharding as a separate change.
• Misleading legacy metadata name: retain the functioning persisted shape to avoid a broad migration; document it in the implementation comment and cover OpenAI pricing behavior with a regression test.

Rollback

Removing the OpenAI structured-system transform and the Chat Completions key extension restores today's automatic-only behavior. Existing Responses cache keys, Anthropic cache control, persisted usage, and pricing remain intact, so rollback requires no data migration.

---

Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh • Cost: $205.62

[ThomasK33]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 102c571002

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".