Skip to content

fix(txn): retry ERR_TRY_AGAIN on the same transaction (native in-place reset) - #1823

Merged
kriszyp merged 4 commits into
mainfrom
kris/tryagain-align
Jul 23, 2026
Merged

fix(txn): retry ERR_TRY_AGAIN on the same transaction (native in-place reset)#1823
kriszyp merged 4 commits into
mainfrom
kris/tryagain-align

Conversation

@kriszyp

@kriszyp kriszyp commented Jul 15, 2026

Copy link
Copy Markdown
Member

Pairs with HarperFast/rocksdb-js#710. rocksdb-js 2.5.0 is now published; rebased onto latest main (which had independently picked up the same dependency bump) and verified against the real native build — see Validation below.

Problem

#1696 fixed the #1695 source-apply ERR_TRY_AGAIN spin by replaying the writes onto a fresh RocksTransaction, because the native layer left the stranded snapshot in place so recommitting the same transaction never converged. But that workaround leaned on a rocksdb-js defect: the transaction-log publish gate (!IsBusy) published the change-feed entry on the failed commit, and the fresh replay (marked isRetry) relied on that premature publish to keep the entry from being lost — an entry visible ahead of its data for every reader in the window, and a permanent phantom if the retry was ultimately abandoned.

Fix

rocksdb-js#710 makes the native layer reset the transaction onto a fresh snapshot on a failed TryAgain commit — exactly as it always did for IsBusy — and publish log entries only on a real commit. So the fresh-transaction replay becomes both unnecessary and wrong (a fresh transaction with isRetry would never publish the now-unpublished entry, re-manifesting #1695). This PR:

  • Recommits the SAME transaction on ERR_TRY_AGAIN, like ERR_BUSY always did. Its committedPosition survives the native reset (WAL write-once, rocksdb-js#668) and its onCommit hook stays attached, so the staged change-feed entry publishes exactly once, only when the retry really commits.
  • Aborts the native transaction on coordinated-retry give-up (pre-existing gap surfaced by review: the RETRY_NOW exhaustion path threw without releasing the handle, leaving an unpublished log position pinning the committed-read watermark until GC; the rejection-path give-up already aborted).
  • Poisons and aborts the whole linked chain on give-up, not just the head, via a shared abortChainAfterRetries() helper mirroring abortDueToTimeout()'s two-pass pattern — and guards each link's wrapper cleanup (txn.abort(), which can throw from savedBlobs/getEntry()) so one link's failure can't strand later links' native handles/read snapshots.
  • Bumps @harperfast/rocksdb-js to ^2.5.0 — the paired-fix floor.

The Table.ts appendedAuditEntry guards from #1696 are deliberately kept: they already govern the ERR_BUSY same-transaction recommit, and the scenario they defend against is less reachable now (failed attempts never publish). Whether the TryAgain-specific rationale is now dead code deserves its own analysis — follow-up issue to come.

Validation

Against the real, published rocksdb-js 2.5.0 build:

  • unitTests/resources/sourceApplyConflictRetry.test.js — including the real compact()-induced ERR_TRY_AGAIN repro, with the assertion inverted from fresh-transaction-id to same-transaction-id — plus a new focused unit test forcing one chain link's wrapper cleanup to throw and asserting later links are still detached/untracked/natively aborted. 5/5 passing.
  • npm run test:unit:resources: 1249/1249 passing.
  • npm ci, format:check, lint:required: clean.
  • CI: Format Check, Lint, Integration Tests, Claude PR Review all green. Unit Test had one unrelated flaky failure (caching.test.js, timing-based) that passed on rerun. The advisory, non-blocking Next.js Integration Tests (downstream) check fails on a pre-existing main type error (unrelated to this diff, already being fixed on a separate branch).

Review history

Two review passes, both addressed:

Cross-model review (Codex ×2 + Harper-domain adjudication; Gemini leg failed — agy hang):

  • Blocker (addressed): version coupling not encoded → ^2.5.0 floor.
  • Verified & fixed: coordinated give-up abort (above).
  • Adjudicated no-new-hazard: the appendedAuditEntry double-apply scenario requires a failed attempt's entry to be visible during retry — exactly what the new publish gate removes.

Live discussion review (Codex + Harper domain pass) after 2.5.0 published:

  • Blocker (addressed): lockfile still resolved 2.4.0 after the manifest bump; regenerated against the real 2.5.0 publish and rebased onto main's independent bump of the same dependency to restore a clean merge (which had also silently stalled pull_request-triggered CI).
  • Significant (addressed): abortChainAfterRetries()'s per-link wrapper txn.abort() was unguarded — a throw on an early link (from savedBlobs/getEntry()) exited the cleanup loop before later chain links were touched, contradicting the two-pass design's own guarantee (also independently flagged by Gemini earlier in review — see inline thread). Fixed by guarding that call too, with a regression test.

KrAIs, via Claude

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates @harperfast/rocksdb-js to version 2.5.0 and refactors transaction retry logic in DatabaseTransaction.ts so that both ERR_BUSY and ERR_TRY_AGAIN errors recommit the same native transaction, which is now reset in place onto a fresh snapshot. It also adds transaction aborting when retries are exhausted to prevent handle leaks. The reviewer recommends that when aborting a chain of linked transactions, all links in the chain should be poisoned and closed first before aborting each individual link to avoid leaking native handles for other transactions in the chain.

Comment thread resources/DatabaseTransaction.ts Outdated
@claude

claude Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

kriszyp and others added 4 commits July 21, 2026 21:20
…e reset)

Pairs with rocksdb-js "align ERR_TRY_AGAIN with the IsBusy reset path".

#1696 worked around a stranded-snapshot ERR_TRY_AGAIN by replaying the writes
onto a *fresh* RocksTransaction, because the native layer left the stranded
snapshot in place so recommitting the same transaction spun forever. That fix
also leaned on rocksdb-js publishing the change-feed entry on the failed commit
(the fresh replay carries isRetry and never re-stages it).

rocksdb-js now resets the transaction onto a fresh snapshot on a failed
TryAgain commit — exactly as it always did for IsBusy — and defers the log
publish until a real commit. So the fresh-transaction replay is both
unnecessary and wrong: a fresh transaction with isRetry would never publish the
now-unpublished entry, re-losing the change-feed entry (#1695). Recommit the
SAME transaction instead, like ERR_BUSY: its committedPosition survives the
reset (WAL write-once) and its onCommit hook stays attached, so the staged
entry publishes exactly once, only when the retry commits.

Updates the regression test to assert the retry reuses the same transaction id
(reset in place) rather than running on a fresh one; the real compact()-induced
ERR_TRY_AGAIN, commutative-increment, and co-batched-duplicate cases all still
converge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The RETRY_NOW exhaustion path threw ServerError without releasing the native
transaction, leaving the handle — and, for a logged transaction, its
unpublished transaction-log position pinning the committed-read watermark —
lingering until GC. The parallel ERR_BUSY/ERR_TRY_AGAIN rejection give-up
already aborts; mirror it here. Pre-existing (not introduced by the TryAgain
alignment), surfaced by cross-model review of this function.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two coordinated-retry give-up paths (RETRY_NOW exhaustion and the
ERR_BUSY/ERR_TRY_AGAIN rejection fallback) aborted only the head's native
transaction, leaving every this.next link holding its native handle and read
snapshot until GC. Mirror abortDueToTimeout's two-pass approach via a shared
abortChainAfterRetries helper: poison every link (open = CLOSED) first, then
abort each link's native transaction and run DatabaseTransaction-level cleanup,
so a throw while aborting one link can't strand the rest. Clearing each link's
native handle + read-snapshot bookkeeping before the per-link abort() prevents a
double native abort (which throws) and abort()'s doneReadTxn loop from spinning
on a nulled handle. The head's blobs are now released too (the old path skipped
them).

Co-Authored-By: Claude Opus <noreply@anthropic.com>
The two-pass design detaches/natively-aborts each link before calling its
DatabaseTransaction-level abort(), specifically so one link's failure can't
strand the rest. But that wrapper abort() call was itself unguarded: it
synchronously walks savedBlobs and calls write.store.getEntry(), which can
throw (closed store, decode error). A throw there exited the loop early,
leaving every later link's native handle/read snapshot un-aborted and still
tracked despite already being detached from the DatabaseTransaction object —
exactly the partial-chain leak the two-pass comment says it prevents.

Catch and log per link, mirroring abortDueToTimeout()'s established pattern,
so cleanup always completes the full chain. Adds a focused unit test that
forces one link's cleanup to throw and asserts later links are still
detached, untracked, and natively aborted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the kris/tryagain-align branch from c94a952 to e0e78f0 Compare July 22, 2026 03:26

@cb1kenobi cb1kenobi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kriszyp
kriszyp merged commit cbcda2b into main Jul 23, 2026
55 of 61 checks passed
@kriszyp
kriszyp deleted the kris/tryagain-align branch July 23, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants