fix(models): retry transient backend failures with backoff; actionable @embed errors (#1594)

[heskew]

Closes #1594 (consolidates #1595, closed as duplicate).

Two resilience/DX gaps in the @embed / model-backend path, both found while dogfooding bulk embedding: a single transient provider error failed the whole write (no retry anywhere in the stack), and the surfaced error gave no hint of provider, model, or cause.

Retry with backoff (all model callers benefit: @embed, scope.models, the /v1 gateway)

New shared fetchWithRetry in resources/models/backendHelpers.ts, wired into the #post path of the ollama, openai, and anthropic backends:

• Retriable: HTTP 408/429/5xx (includes Anthropic's 529 overloaded) and transient network rejections. Deterministic client errors (400/401/404/…) are never retried.
• Retry-After (delta-seconds or HTTP-date) is honored up to a 30s cap — a server demanding a longer wait gets its response surfaced immediately rather than retried early (impolite) or held open (stalls the write).
• Backoff: jittered exponential — retryBackoffMs base (default 500ms), doubling per attempt, 10s per-sleep cap, jitter factor in [0.5, 1).
• Budget: every attempt and backoff sleep runs under the existing composed signal, so requestTimeoutMs keeps its documented meaning as the overall per-call deadline — retries fit inside the budget rather than extending it. Aborts (caller or timeout) cancel a pending backoff sleep immediately and are never retried.
• Config (per model entry): maxRetries (default 2, 0 disables) and retryBackoffMs. Malformed values fall back to defaults.

bedrock goes through the AWS SDK, which owns its retry loop — the same maxRetries knob is plumbed to the SDK's maxAttempts (maxRetries + 1) instead of double-wrapping. Default matches the SDK's own (3 attempts). No retryBackoffMs there; the SDK's retry strategy manages delays.

Design decisions (answering the questions consolidated from #1595)

• Layer: backend layer, not Models dispatch — so every caller benefits and per-backend semantics (SDK-managed vs fetch) stay local.
• Failover interaction: a backend exhausts its own retry budget first, then Models.ts multi-candidate failover proceeds to the next candidate exactly as today.
• Streaming: retry covers connection/status failures before the body is handed over. A mid-stream failure after chunks have been yielded is deliberately not retried (partial results were already delivered).
• Batching/queueing @embed bulk writes: out of scope; separate design if pursued.

Actionable @embed error surface

#1593 already added the error class name + upstream HTTP status to the sanitized rethrow. This completes it:

• The configured model name is now always included.

• The backend's message is surfaced verbatim — but only when the error is a ServerError subclass carrying upstreamStatus, which is exactly our backend classes' HTTP-error path whose messages are deliberately sanitized and length-capped (no request content, capped upstream text). Example:

  Failed to compute embedding for attribute "embedding" (model "default") [OpenAIBackendError] (backend HTTP 404): OpenAI /embeddings returned HTTP 404: models/default is not found

• A bare upstreamStatus on a non-ServerError is not proof of sanitization (custom embedder errors can carry anything): those keep the @embed / models.embed: logical model name is forwarded as the wire model id (404 when they differ), and the error is unhelpful #1593 posture — safe identifiers only, — see server log for details. There's a regression test pinning this.

• The ServerError check is lazily required in the failure path only, mirroring the existing getLogger pattern (hdbError imports the logger; fail-closed to the sanitized message if the require fails).

Tests

• backendHelpers.test.js: 30+ new cases — resolveRetryConfig validation, retriable-status matrix, Retry-After parsing (delta/date/garbage/past), jitter bounds + cap, abortableSleep abort semantics, and fetchWithRetry scenarios (retry-then-success, Retry-After honored/over-cap, non-retriable passthrough, exhaustion, maxRetries: 0, network-error retry/exhaustion, abort-never-retried, abort-during-backoff).
• openai/index.test.js: retry through the real #post (429 → success), maxRetries: 0 surfaces upstreamStatus, non-retriable 400 single-attempt.
• bedrock/index.test.js: maxAttempts plumbing (default → 3, maxRetries: 0 → 1).
• embedHook.test.js: verbatim surface for real backend-shaped errors (message + model + status + class), spoofed upstreamStatus on a plain Error stays sanitized.

All 249 tests across the six affected suites pass; oxlint and prettier clean; build shows only the known pre-existing TS errors in unrelated files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BdbdepRFuyUWoEmNN1pSVQ

---

Review notes (HEG step 10 — cross-model review, 2026-07-17)

Harper-domain pass (deep-review + adjudication): no blockers. Direction-fit verified: aligns with the bounded-backoff precedents (DatabaseTransaction commit retry, #1371 index-backfill retry), preserves the composeSignal/requestTimeoutMs contract, extends the #1593 sanitized-error posture deliberately (fail-closed gate; provider text capped at 500 chars), and is orthogonal to the uWS/h2c/h3 server-side networking work (outbound fetch only).

Open items for the human reviewer (found by the review, not resolved in-PR):

1. Lock-hold amplification on the cache-from-source path. Table.ts holds the per-record store lock while embedBefore() runs; that path's sourceContext need not carry a signal, and with no requestTimeoutMs configured the retry budget can stretch the hold (worst case ~60s+ under a rate-limited provider honoring Retry-After). Pre-existing exposure — the network call already ran under that lock unbounded — but retries amplify the worst case. In-PR mitigations: maxRetries is now clamped at 10, and setting requestTimeoutMs per model entry bounds it fully. A structural fix (bounding total embed time on the lock-held refresh path in Table.ts) is out of this PR's scope — flag if you want it here or as a follow-up.
2. Network-error retry is broader than "transient" — any non-abort rejection retries, including deterministic ones (down endpoint, NXDOMAIN), adding maxRetries × backoff latency before a clear misconfiguration surfaces. Deliberate: distinguishing transient from permanent rejection shapes across undici versions is fragile, and the cost is bounded (~1.5s at defaults).

Docs decision (HEG step 16): maxRetries/retryBackoffMs are user-facing model-entry config, but models config has no docs pages yet anywhere — documentation lands wholesale with the #510 docs arc, so no companion docs PR here.

Caveats: both outside-model review legs failed this run (codex CLI init error; agy degraded then timed out) — this was a domain-pass-only review. Re-run the outside legs if you want second-model corroboration before merge.

[gemini-code-assist]

Code Review

This pull request implements bounded retry logic with jittered exponential backoff and Retry-After header support across several model backends (Anthropic, Bedrock, Ollama, and OpenAI), and refines embedding error surfacing to prevent sensitive data leaks. The feedback suggests improving robustness by explicitly checking for trimmed empty strings before numeric coercion of the Retry-After header, and wrapping property accesses on untrusted error objects in try/catch blocks to prevent potential crashes from optional chaining.

[claude]

Reviewed; no blockers found.