fix(core): retry model capacity errors

[steipete-oai]

Summary

Treat structured model-capacity failures as a bounded, recoverable state instead of ending the turn immediately.

Normal sampling and remote compaction v2 now keep the same turn alive for up to three patient retries. The retry waits have 30-second, 2-minute, and 5-minute minimums, each with positive jitter so clients never retry earlier than the base schedule or synchronize on the same instant.

Why

The Responses API reports temporary model capacity as HTTP 503 with either server_is_overloaded or slow_down. Previously these responses were consumed by the ordinary fast transport retry layer. Once that short budget was exhausted, an otherwise healthy long-running Codex task stopped and required the user to notice and continue it manually.

Capacity is different from a generic transport failure:

• Switching from WebSocket to HTTP does not relieve backend capacity.
• Immediate retries add load while the selected model is already saturated.
• A long-running task benefits more from patient recovery than from failing quickly.
• The retry must remain cancellation-aware so the user can still interrupt the turn.

Retry behavior

• Recognize only HTTP 503 responses whose structured error code is server_is_overloaded or slow_down.
• For normal sampling and remote compaction v2, return those responses directly to the turn-level retry loop instead of retrying them in the fast HTTP transport layer.
• Retry the same turn up to three times after the initial failure.
• Wait at least 30 seconds, 2 minutes, and 5 minutes. Each delay receives 0–100% positive jitter, so cumulative waiting is 7.5 minutes to just under 15 minutes.
• Keep capacity retries on a separate counter from ordinary stream retries. A disconnect or generic 5xx therefore cannot consume or multiply the capacity budget.
• Do not invoke WebSocket-to-HTTP fallback for capacity errors; ordinary retry and fallback behavior remains unchanged.
• Emit StreamError events with ServerOverloaded and visible Reconnecting... 1/3, 2/3, and 3/3 progress.
• Allow cancellation while opening the request, consuming the response stream, or waiting in backoff.

Compaction and input safety

Remote compaction v2 uses the same capacity policy as sampling, including its independent budget and cancellation behavior.

Pre-turn compaction runs before the incoming user input is recorded in thread history. If capacity remains exhausted at that point, the protocol now reports serverOverloadedBeforeInput rather than the ordinary serverOverloaded variant. Clients can use that distinction to replay the saved request payload instead of sending an empty continuation or duplicating input.

The new error variant is propagated through the core protocol, app-server protocol, generated TypeScript types, and JSON schemas. Both overload variants retain the same rate-limit/capacity presentation in the TUI.

When previous-model compaction is rejected and Codex falls back to the current model, a capacity error or cancellation from the fallback attempt is preserved instead of being hidden behind the original rejection.

Intentional non-changes

• Generic HTTP 5xx, transport, and stream retry policies are unchanged.
• Local compaction retains its existing provider-level retry behavior.
• Guardian continues to own its deadline-bound retry policy; this change does not nest the long capacity schedule inside Guardian review.
• Codex does not silently switch to a lower-intelligence model.
• After the three retries are exhausted, the error is still terminal for this turn. Continued goal-level requeueing is separate work in Retry goals after model capacity errors #31176.

The large file count is primarily exhaustive regression coverage plus regenerated protocol schemas.

Test plan

• Verified the overload classifier accepts only the two structured capacity codes on HTTP 503.
• Verified capacity retries use a separate same-turn budget after ordinary transport and stream retries.
• Verified all three retry progress events are emitted.
• Verified remote compaction v2 can recover after exhausting its ordinary stream budget.
• Verified Guardian does not inherit the long capacity retry policy.
• Verified manual remote compaction can be interrupted while a response is pending.
• Verified local compaction keeps provider-owned retry behavior.
• Verified pre-turn local and remote compaction report uncommitted input correctly.
• Verified successful pre-turn recovery records incoming input exactly once.
• Verified previous-model fallback preserves terminal capacity errors.
• codex-protocol and codex-app-server-protocol: 492 tests passed on the rebased head.
• Focused pre-turn capacity suite: 3 tests passed across local and remote compaction exhaustion/recovery.
• Guardian transient-retry regression passed.
• Schema fixtures regenerated cleanly.
• just fix -p codex-core -p codex-protocol -p codex-app-server-protocol -p codex-tui passed.
• just fmt passed.

What users see

When the selected model is temporarily at capacity, Codex remains active and retries automatically instead of immediately ending a long-running task.

Related: #31176, #22390, #31278.

[cowwoc]

@steipete-oai Instead of retrying after 30 second, 2 minutes, 5 minutes and then giving up, can you please implement a proper exponential backoff that starts with 3 seconds and maxes out at 5 minutes? The number of retries should be large enough to prevent overnight goals from aborting.

[wakamex]

A bounded default retry capacity makes sense for interactive sessions, but goal-level requeueing, as proposed in #31176, does not help long-running harnesses that do not use /goal.

Could users configure the maximum number of capacity retries, including an unlimited option for unattended harnesses? Otherwise, every harness must implement special handling to restart turns that end because the selected model is temporarily at capacity.

Unlimited retries could continue at the five-minute base delay with positive jitter, remain cancellation-aware, and use a separate budget from ordinary transport retries.

For example:

[responses.capacity_retries]
max_retries = 3

Or:

[responses.capacity_retries]
max_retries = "unlimited"

[relh]

consideration worth noting:

This queue issue here is a big one because often it will break the prompt with this capacity issue presumably from a load balancer but then it will cascade right into the follow up tab queued prompt even though that prompt is/can be dependent on the previous prompt having finished correctly

for example, if ive prompted "implement and merge A which fixes B" and then tab queued up "now that B is fixed we can implement C" then "selected model is at capacity" hits, it can immediately halt our finishing of A but then because a load balancer switches us to somewhere where the model is not at capacity, it can from a dirty halfdone feature A begin the confused implementation of C from a partially done conversation.