Claim cursors: graveyard-resistant polling on PostgreSQL

[jpcamara]

Solves the PostgreSQL dead-tuple pathology: long-running queries pin the xmin horizon, vacuum can't reap solid_queue_ready_executions, and every poll re-scans the graveyard at the low end of the polling index — CPU proportional to how long the horizon stays pinned. Claim cursors let polls seek past the graveyard instead of scanning through it.

Design

One claim cursor per queue key: each process remembers the last (priority, id) position it observed. Fast polls make a single lexicographic row-constructor seek — WHERE (priority, id) > (?, ?) ORDER BY priority, id — which descends the polling index past the entire graveyard in one shot, across all priorities. When a seek comes up empty, the position is cleared and polls skip the claim query entirely until discovery is due.

Discovery is fundamental, not a safety net: the cursor is not a lower bound for live work — a competing claim of a lower id can roll back after this process advanced past it, sequences aren't commit-ordered so a row can allocate a lower id and commit late, and an empty SKIP LOCKED seek can mean the remaining rows were merely locked by a peer. Cursor-free discovery therefore runs on a time-based cadence and comes in two forms:

• Floored pass (claim_cursors_discovery_interval, 1s default): claims only id > the sequence watermark recorded before the last full pass, so its cost tracks recent throughput instead of graveyard size. Every newly enqueued row lands above the watermark, so pickup latency for fresh work — including strictly-higher-priority arrivals below the cursor — stays bounded by the short cadence at any polling interval.
• Full pass (claim_cursors_full_discovery_interval, 10s default, jittered downward so co-booted workers don't scan in lockstep; 0 disables the split): the only pass that reaches rows at or below the watermark (rollback overshoot, late commits), and the only one allowed to move a cursor, healing stragglers within one full interval.

The watermark is trusted only where its guarantee holds — PostgreSQL 10+, an owned id sequence with CACHE 1 — and degrades to all-full passes otherwise. Floored claims run with sorts disabled inside a savepoint (settings restored before release, reverted by rollback) because pinned-horizon statistics make every plan cost-tie and the tie can land on a filtered legacy-index scan that re-reads the graveyard the floor exists to skip.

Honest framing: discovery still scans the graveyard once per cadence, so polling is graveyard-resistant, not dead-tuple-immune — the cost is amortized to O(graveyard/cadence) rather than eliminated. The other trade: a strictly-higher-priority arrival waits for the next discovery (≤ ~1s), inside the slack FOR UPDATE SKIP LOCKED already implies.

Ordering change (deliberate): within a priority, claims follow readiness order (id) instead of enqueue order (job_id). Observable difference: a scheduled job coming due queues behind already-waiting same-priority jobs rather than jumping ahead of them. Polling indexes move to (priority, id) / (queue_name, priority, id); the README includes the migration for existing installs.

PostgreSQL only: row-constructor index seeks and the motivating pathology are PostgreSQL-specific. Other adapters keep the classic path untouched. config.solid_queue.claim_cursors = false disables.

Benchmark

Pinned horizon, 200k unremovable dead tuples in ready_executions (confirmed via VACUUM VERBOSE, heap decorrelated from id order), real ReadyExecution.claim path:

	buffer hits	time
classic poll (no cursors)	200,303	53.3 ms
full discovery pass (unbounded)	200,283	49.1 ms
floored discovery pass	982	9.8 ms
fast poll (cursor seek)	~1	0.9 ms
200-poll loop, classic	4,005,860	1,009 ms
200-poll loop, cursors + bounded discovery	418,406	360 ms

The floored pass costs 204x fewer buffer hits than the scan it replaces; the whole polling loop touches 95.7x fewer buffers than pre-cursor polling. Because the frequent pass is cheap and sees all new work, this holds at any polling_interval — including fleets that poll at 1s, where the short cadence coincides with every idle poll.

Testing

• ClaimCursorsTest (PG-gated) asserts query counts and shapes via a sql.active_record capture: single-seek claiming across priorities, instant same-priority pickup, empty seek clearing the position and suppressing queries, higher-priority arrivals found once discovery is due (deterministic expire_discovery!, no sleeps), overshoot healing via discovery, readiness-order semantics, flag-off classic behavior
• Full suite green on PostgreSQL with cursors live everywhere; SQLite and MySQL green on the classic path
• Tests reset cursor state in the shared setup — process-local state must not outlive the records it was derived from

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.