feat: harden LLM API fault tolerance against 429 and overload - #1530
Conversation
- retry more transient errors: 408/409/429/5xx/529, an embedded upstream status_code=429 in OpenAI Responses stream errors, and unclassified provider errors as a last-resort fallback - honor server Retry-After (parsed into APIStatusError.retryAfterMs by the OpenAI and Anthropic providers); chatWithRetry prefers it over its backoff - align app-level backoff with claude-code (500ms base, 32s cap, factor 2, up to 25% jitter) so high-attempt configs ride out multi-minute overload - emit a turn.step.retrying meta line in -p --output-format stream-json
🦋 Changeset detectedLatest commit: 5205322 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ef159dc99
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // (400/401/403/404/422) and the recovery-owned context-overflow / | ||
| // request-too-large subclasses keep their dedicated handling instead of | ||
| // burning retries first. | ||
| return error instanceof ChatProviderError; |
There was a problem hiding this comment.
Because ChatProviderError is also used for deterministic preflight/serialization failures (for example missing API keys in requireProviderApiKey and unrecognized OpenAI Responses stream errors in errorFromOpenAIResponsesEvent), this fallback now makes non-transient 4xx/config errors retryable. With a high max_retries_per_step, the new 32s-capped backoff can delay surfacing bad auth or bad-request failures for minutes, contrary to the intended deterministic-4xx behavior; use a distinct transient wrapper or status-gate the gateway fallback instead of retrying every base provider error.
Useful? React with 👍 / 👎.
| // Mirrors claude-code's retryable statuses: 408 (request timeout), 409 | ||
| // (lock/conflict timeout), 429 (rate limit), 5xx (server errors) and 529 | ||
| // (provider overloaded — the "engine is currently overloaded" case). | ||
| return [408, 409, 429, 500, 502, 503, 504, 529].includes(error.statusCode); |
There was a problem hiding this comment.
The new behavior is documented here as retrying 5xx, but the predicate still enumerates only 500/502/503/504 plus 529. Providers and gateways commonly use other server-side 5xx statuses such as 520, 522, or 524; those still become APIStatusError instances and will fail immediately instead of entering the retry path this change is adding.
Useful? React with 👍 / 👎.
1 participant
Related Issue
No linked issue — see Problem.
Problem
In long-running eval runs, a kimi-code run could die mid-rollout when the provider returned a 429 / overload / stream interruption, surfacing as an unfinished turn ("API error occurred at step N") and triggering far more container-level retries than a peer harness against the same model and backend. The OpenAI path (a) did not retry several transient errors, (b) used a small backoff budget that gave up long before an overloaded engine recovered, and (c) ignored the server
Retry-After, whereas that peer harness absorbs these in place.What changed
status_code=429in OpenAI Responses stream error text is treated as a rate limit; and any unclassified provider error is retried as a last-resort fallback instead of failing the run on the first error.Retry-Afterand align the backoff with the reference CLI: the OpenAI and Anthropic providers parse the responseRetry-Afterheader into the error and the retry loop prefers it over its own delay; the app backoff is now 500ms base / 32s cap / factor 2 / up to 25% jitter, so high-attempt configs ride out multi-minute overload.-p --output-format stream-jsonnow emits aturn.step.retryingmeta line so consumers can observe retries.Request timeout is unchanged (the OpenAI SDK default of 600s already matches the reference CLI's default). Deterministic 4xx (auth, not-found, bad-request) are intentionally still not retried.
Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.