Bound the claim cursor discovery pass with a sequence watermark

[jpcamara]

Stacked on claim-cursors.

Claim cursors made fast polls cheap. This bounds the cost of the discovery pass that keeps them correct — the one term that still grows without bound while the xmin horizon is pinned.

The problem

ReadyExecution.claim on the base branch has three query shapes. Measured on the PostgreSQL system that motivated cursors, with the xmin horizon pinned by long-running queries so vacuum cannot reap claimed ready executions:

	buffer hits	time
Today's poll (no cursors)	250,794	114.3 ms
Discovery pass (new index, no cursor bound)	202,589	53.5 ms
Fast poll (cursor seek)	3,442	1.8 ms

Discovery is ~77% of what polling still costs, and it is the only term bounded by nothing: the fast poll's cost is bounded by throughput × polling interval, while discovery scans from the low end of the (priority, id) index — straight through the graveyard — every time it runs.

Discovery can't simply be dropped or slowed down. It exists because a cursor is not a lower bound for live work:

1. Rollback overshoot. A row locked by another process's claim transaction is skipped under SKIP LOCKED, this process advances past it, and that transaction then rolls back — the row is live again, below the cursor, where no fast seek will ever look.
2. Late commits. PostgreSQL sequences are not commit-ordered, so a row can be allocated a lower id and commit after the cursor has advanced past a higher one.

Discovery also does routine work: it finds strictly-higher-priority arrivals, which land below the cursor because their priority sorts first.

The design

Split those two jobs apart, because only one of them is rare.

A floored pass runs on the existing short cadence (claim_cursors_discovery_interval, 1s). Its floor is the solid_queue_ready_executions sequence high-water mark, read immediately before the last unbounded pass's claim query. Every row allocated since then has a greater id, at every priority, so id > floor still finds higher-priority arrivals without rescanning the graveyard buried underneath them.

An unbounded pass runs on a new long cadence (claim_cursors_full_discovery_interval, 10s by default), records the next floor, and remains the only pass that can reach a row sitting at or below the floor. Setting the interval to 0 makes every discovery pass unbounded — byte-for-byte the base branch's behavior.

Two rules keep the floored pass sound:

• It claims only (priority, id) <= cursor while the fast seek claims (priority, id) > cursor in the same poll. Disjoint regions, discovery region first, so claims keep their global (priority, id) order.
• It never advances or clears the cursor. It may only seed one that an empty seek cleared. Only the unbounded pass — the one query allowed to see everything below where the cursor lands — may move it.

The floor is only recorded where its guarantee actually holds: PostgreSQL 10+ (whose pg_sequence catalog exposes the cache setting), sequence cache of 1 (PostgreSQL's default), and a table that owns its sequence. Anything else records no floor and keeps every pass unbounded — degradation, never an error. The cache is re-checked at every unbounded pass, so ALTER SEQUENCE ... CACHE at runtime self-heals in both directions.

Cursor state (positions, floors, deadlines) lives in a registry keyed by connection pool: these are positions in one table's id space and must not leak between datastores a process might poll. The full-pass deadline is jittered only downward (interval - rand * 0.25 * interval), so worker fleets don't scan the graveyard in lockstep and the configured interval stays a hard upper bound.

A plan-stability fix the benchmark caught

The base branch's additive schema keeps the legacy (priority, job_id) polling indexes alongside the new (priority, id) ones. Under pinned-horizon statistics — zero visible rows, so every plan estimates rows=1, cost≈8.3 — the planner's choice for the floored query is a cost tie, broken arbitrarily. At 200k dead tuples it picked the legacy index with Filter: (id > floor): a heap fetch per dead tuple, 200,277 buffer hits, plus a Sort — the exact scan the pass exists to skip. Measured, not theoretical, and nondeterministic (a 40k graveyard chose well, 200k chose badly).

A floored query only ever wants a polling index walked in (priority, id) order; a sort-based plan can only re-read the graveyard. So the two floored claims run with sort plans off (SET LOCAL enable_sort, plus enable_incremental_sort where it exists) — inside a savepoint, with the previous settings captured first and restored before the savepoint releases, and reverted by PostgreSQL itself if it rolls back. The settings cannot escape a caller's transaction, other adapters and the classic path never reach this, and un-migrated schemas keep working because PostgreSQL treats disabled plan types as cost-penalized, not impossible. Disabled plans cannot tie with enabled ones, so the outcome is deterministic:

Index Scan using index_solid_queue_poll_all_by_id
  Index Cond: ((ROW(priority, id) <= ROW(0, 2505)) AND (id > 2540))
  Buffers: shared hit=1029          # legacy-index sort plan: shared hit=200277

Why no job can be stranded

Let F be the floor and P the cursor. Take any live, unclaimed row R = (p, i).

• If (p, i) > P, the fast seek covers it.
• If i > F, the floored pass covers it — below P via the below-cursor sweep, and across all priorities via the seeding sweep when no cursor exists.
• Otherwise R is below the cursor and at or below the floor. F was read before the last unbounded pass's snapshot, so R's id was already allocated then; for R to be live and unclaimed now, it must be a rollback overshoot or a late commit. The unbounded pass reaches it within one (jittered-downward) full interval plus one short interval.

So the floored pass loses nothing relative to unbounded discovery except healing those two anomalies; it introduces no new class of missed row. Supporting liveness facts: deadlines are recorded only after the claim transaction commits, so a raise leaves discovery due; a fresh or restarted process has no floor, and a nil floor forces an unbounded pass; a paused queue's key comes back with its deadlines expired; a replacement connection pool starts with no state and full-discovers on its first poll.

Designs rejected

A sound lower bound derived from PostgreSQL transaction state (pg_snapshot_xmin / backend_xmin plus a sequence watermark promoted once old xids resolve) — negative result. It needs an xid→sequence-id mapping PostgreSQL does not guarantee: xids are assigned at first write, nextval needs no xid, and a BEFORE INSERT trigger can widen the gap arbitrarily. It would make CACHE 1 a correctness requirement rather than a degradation trigger. And fatally, it only bounds newly visible rows — a rolled-back claim resurrects a row committed long ago, which no xid-derived floor excludes. Sol (consulted at design time) confirmed the negative result independently.

An incremental bounded sweep below the cursor. Slices must be key ranges (a row-count bound teaches you nothing when the region is pure graveyard), so per-pass cost is proportional to slice size — measured 3,625 buffer hits per 10k-id slice — and worst-case healing latency becomes graveyard_size / slice_size × interval, which grows as the pathology worsens. A timed unbounded pass keeps that latency a fixed constant instead.

Deriving the floor from the highest id claimed (the initial draft — rejected in design review). Because claims are ordered by (priority, id), the highest id a pass claims is not an id it has swept past: with cursor (5,5000) and arrivals A=(1,10000), B=(2,6000) at limit 1, claiming A and floating the floor to 10000 hides B below both cursor and floor. The test successive floored passes reach every arrival below the cursor fails if that variant is reintroduced — verified. The same review is why floored passes never move the cursor: advancing it from an id-filtered subset skips rows the filter hid.

Benchmark

Synthetic pinned-horizon graveyard: 200,000 dead tuples confirmed "dead but not yet removable" by VACUUM VERBOSE, four-index (additive) schema, measured through the real ReadyExecution.claim path with pg_stat_database buffer deltas. "Base" is claim_cursors_full_discovery_interval = 0, i.e. the base branch's behavior exactly.

The heap is deliberately decorrelated from id order, as a long-lived table's is (concurrent inserters each fill their own target pages). On a pristine append-only table an unbounded scan costs ~1 buffer per heap page; decorrelated it costs ~1 buffer per dead tuple — the regime the production numbers show. Measured 0.96 buffers per dead tuple, so the harness reproduces production's cost structure.

	buffer hits	time
classic poll (no cursors)	200,303	53.3 ms
base discovery pass (unbounded)	200,283	49.1 ms
bounded discovery pass (floored)	982	9.8 ms
fast poll (cursor seek)	~1	0.9 ms
base 200-poll loop	4,005,860	1,009 ms
bounded 200-poll loop	418,406	360 ms

Discovery pass: 204x fewer buffer hits, 5x faster. A 200-poll loop at Solid Queue's default cadences (0.1s polling, 1s discovery, 10s full discovery): 9.6x fewer buffer hits, 2.8x faster — and 95.7x fewer hits than the pre-cursor poll. The bounded loop's remaining cost is almost entirely its two unbounded passes; everything between them is bounded by enqueue volume, not graveyard size.

The tradeoff

Rollback overshoot and late commits now heal within the full interval (plus the short cadence it piggybacks on) instead of within the short interval — roughly 10s instead of 1s at the defaults. Normal work, including strictly-higher-priority arrivals, keeps its ≤1s pickup latency. The healing latency is a fixed, configurable constant, independent of graveyard size — the property the incremental sweep could not offer. Raise the interval to spend less CPU while a horizon is pinned; lower it to heal faster; 0 restores base behavior.

The unbounded pass is still O(graveyard) when it runs — its cost is amortized 10x and jitter-spread across the fleet, not eliminated.

Review

Designed and reviewed in four rounds with a second model (GPT-5.6 "Sol", xhigh reasoning), which: confirmed the transaction-state lower bound is unsound, rejected the claim-following watermark with the counterexamples above, prescribed the sequence-watermark + timed-backstop shape, and in implementation review caught the PG-10 catalog dependency, the jitter-direction latency bug, the datastore key collision, the sequence-cache assumption, and the transaction-scope leak in the first version of the planner guard — all fixed and pinned by tests. The plan-stability defect itself was caught by the benchmark.

Verification

• TARGET_DB=postgres bin/rails test test/models/solid_queue/claim_cursors_test.rb test/models/solid_queue/ready_execution_test.rb test/unit/worker_test.rb — 57 runs, 0 failures
• TARGET_DB=sqlite bin/rails test test/models/solid_queue/ready_execution_test.rb — 19 runs, 0 failures (classic path untouched; cursors stay PostgreSQL-only)
• bundle exec rubocop on all changed files — no offenses
• All base-branch ClaimCursorsTest tests pass unchanged. Of the 13 new tests, 9+ fail with the production code reverted to base; the rest pin base-equivalent compatibility modes by construction.

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.