Skip to content

(feat) expontential back-off when encountering rate limit errors #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tibo-openai merged 2 commits into from
Apr 17, 2025
Merged

(feat) expontential back-off when encountering rate limit errors #153

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ceb67f5
w
tibo-openai Apr 17, 2025
1f47299
w
tibo-openai Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions codex-cli/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

72 changes: 49 additions & 23 deletions codex-cli/src/utils/agent/agent-loop.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import OpenAI, { APIConnectionTimeoutError } from "openai";

// Wait time before retrying after rate limit errors (ms).
const RATE_LIMIT_RETRY_WAIT_MS = parseInt(
process.env["OPENAI_RATE_LIMIT_RETRY_WAIT_MS"] || "15000",
process.env["OPENAI_RATE_LIMIT_RETRY_WAIT_MS"] || "2500",
10,
);

Expand Down Expand Up @@ -569,11 +569,6 @@ export class AgentLoop {
);
continue;
}
const isRateLimit =
status === 429 ||
errCtx.code === "rate_limit_exceeded" ||
errCtx.type === "rate_limit_exceeded" ||
/rate limit/i.test(errCtx.message ?? "");

const isTooManyTokensError =
(errCtx.param === "max_tokens" ||
Expand All @@ -597,30 +592,61 @@ export class AgentLoop {
return;
}

const isRateLimit =
status === 429 ||
errCtx.code === "rate_limit_exceeded" ||
errCtx.type === "rate_limit_exceeded" ||
/rate limit/i.test(errCtx.message ?? "");
if (isRateLimit) {
if (attempt < MAX_RETRIES) {
// Exponential backoff: base wait * 2^(attempt-1), or use suggested retry time
// if provided.
let delayMs = RATE_LIMIT_RETRY_WAIT_MS * 2 ** (attempt - 1);

// Parse suggested retry time from error message, e.g., "Please try again in 1.3s"
const msg = errCtx?.message ?? "";
const m = /retry again in ([\d.]+)s/i.exec(msg);
if (m && m[1]) {
const suggested = parseFloat(m[1]) * 1000;
if (!Number.isNaN(suggested)) {
delayMs = suggested;
}
}
log(
`OpenAI rate limit exceeded (attempt ${attempt}/${MAX_RETRIES}), retrying in ${RATE_LIMIT_RETRY_WAIT_MS} ms...`,
`OpenAI rate limit exceeded (attempt ${attempt}/${MAX_RETRIES}), retrying in ${Math.round(
delayMs,
)} ms...`,
);
// eslint-disable-next-line no-await-in-loop
await new Promise((resolve) =>
setTimeout(resolve, RATE_LIMIT_RETRY_WAIT_MS),
);
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
} else {
// We have exhausted all retry attempts. Surface a message so the user understands
// why the request failed and can decide how to proceed (e.g. wait and retry later
// or switch to a different model / account).

const errorDetails = [
`Status: ${status || "unknown"}`,
`Code: ${errCtx.code || "unknown"}`,
`Type: ${errCtx.type || "unknown"}`,
`Message: ${errCtx.message || "unknown"}`,
].join(", ");

this.onItem({
id: `error-${Date.now()}`,
type: "message",
role: "system",
content: [
{
type: "input_text",
text: `⚠️ Rate limit reached. Error details: ${errorDetails}. Please try again later.`,
},
],
});

this.onLoading(false);
return;
}
this.onItem({
id: `error-${Date.now()}`,
type: "message",
role: "system",
content: [
{
type: "input_text",
text: "⚠️ Rate limit reached while contacting OpenAI. Please try again later.",
},
],
});
this.onLoading(false);
return;
}

const isClientError =
Expand Down