Skip to content

refactor: extract dead_letter, archival, worker ops into diesel_common#207

Merged
pratyush618 merged 8 commits into
masterfrom
refactor/diesel-common-expansion
May 28, 2026
Merged

refactor: extract dead_letter, archival, worker ops into diesel_common#207
pratyush618 merged 8 commits into
masterfrom
refactor/diesel-common-expansion

Conversation

@pratyush618

@pratyush618 pratyush618 commented May 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Extract all 4 dead_letter ops into impl_diesel_dead_letter_ops! macro (100% identical between SQLite/Postgres)
  • Extract list_archived into impl_diesel_archival_ops! macro (archive_old_jobs stays backend-specific due to INSERT OR IGNORE vs ON CONFLICT DO NOTHING)
  • Extract 6 of 7 worker ops into impl_diesel_worker_ops! macro (register_worker stays backend-specific due to replace_into vs on_conflict.do_update)

Eliminates ~250 LOC of duplication across 3 file pairs, extending the existing diesel_common/ macro pattern (jobs, locks, logs, metrics).

Test plan

  • cargo test --workspace passes
  • cargo check --workspace --features postgres compiles
  • cargo clippy --workspace clean

Summary by CodeRabbit

  • Refactor
    • Storage operations for job archival, dead-letter queue handling, and worker management now share unified implementations across database backends, reducing duplication and improving consistency.
  • Tests
    • Added SQLite tests covering dead-letter retry error handling and removal of stale workers.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 27, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 38c18511-3f52-4551-a9a3-6d16283a5bbe

📥 Commits

Reviewing files that changed from the base of the PR and between 0162b5f and d86e050.

📒 Files selected for processing (3)
  • crates/taskito-core/src/storage/diesel_common/dead_letter.rs
  • crates/taskito-core/src/storage/diesel_common/workers.rs
  • crates/taskito-core/src/storage/sqlite/tests.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/taskito-core/src/storage/diesel_common/dead_letter.rs
  • crates/taskito-core/src/storage/diesel_common/workers.rs

📝 Walkthrough

Walkthrough

Extracts duplicate Diesel database logic into three macros in diesel_common/ (archival, dead-letter, workers) and replaces explicit Postgres and SQLite implementations with macro invocations; adds SQLite tests for dead-letter retry-not-found and worker reaping.

Changes

Storage Macro Consolidation

Layer / File(s) Summary
Shared Diesel macro implementations
crates/taskito-core/src/storage/diesel_common/archival.rs, crates/taskito-core/src/storage/diesel_common/dead_letter.rs, crates/taskito-core/src/storage/diesel_common/workers.rs
Introduces impl_diesel_archival_ops!, impl_diesel_dead_letter_ops!, and impl_diesel_worker_ops! macros that implement paginated archived-job listing, transactional DLQ move/list/retry/purge, and worker heartbeat/status/list/reap/unregister/claim queries respectively; each macro is re-exported for crate use.
Module declarations and re-exports
crates/taskito-core/src/storage/diesel_common/mod.rs
Adds archival, dead_letter, and workers submodules and re-exports the three macros for storage backends.
Postgres backend refactoring
crates/taskito-core/src/storage/postgres/archival.rs, crates/taskito-core/src/storage/postgres/dead_letter.rs, crates/taskito-core/src/storage/postgres/workers.rs
Replaces explicit Postgres implementations for archival listing, dead-letter ops, and worker ops with impl_diesel_* macro invocations attached to PostgresStorage.
SQLite backend refactoring and tests
crates/taskito-core/src/storage/sqlite/archival.rs, crates/taskito-core/src/storage/sqlite/dead_letter.rs, crates/taskito-core/src/storage/sqlite/workers.rs, crates/taskito-core/src/storage/sqlite/tests.rs
Replaces explicit SQLite implementations with the shared macros and adds tests asserting retry_dead returns QueueError::JobNotFound for missing IDs and that reap_dead_workers removes only stale workers.

🎯 3 (Moderate) | ⏱️ ~20 minutes

"🐰 I stitched three macros in a hop,
Postgres and SQLite now share the top.
Jobs archived, dead letters tried,
Workers reaped where heartbeats died —
Cleaner code, lighter hop!"

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: extracting dead_letter, archival, and worker operations into shared Diesel macros in diesel_common, which aligns with all modified files and the PR's core objective.
Docstring Coverage ✅ Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/diesel-common-expansion

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/taskito-core/src/storage/diesel_common/dead_letter.rs`:
- Around line 82-87: The code in retry_dead currently maps every DB error from
dead_letter::table.find(...).first(&mut conn) to QueueError::JobNotFound and
later doesn't verify the delete actually removed a row, which causes races and
masks real DB errors; change the first() call to match the diesel error: if
Err(diesel::result::Error::NotFound) return
QueueError::JobNotFound(dead_id.to_string()), otherwise propagate or wrap other
diesel errors as a DB error (e.g., QueueError::Db or return Err(err.into())).
For the delete step (the diesel::delete(...).execute(&mut conn) call around
where DeadLetterRow is removed), check the returned affected rows count and if
it is 0 return QueueError::JobNotFound(dead_id.to_string()), and if execute
returns any other DB error propagate/wrap it instead of treating it as
not-found; apply the same pattern to the other similar block around lines
129-136.

In `@crates/taskito-core/src/storage/diesel_common/workers.rs`:
- Around line 53-60: The current logic selects dead_ids from workers::table and
then deletes rows by worker_id alone, which can remove workers that updated
their heartbeat after the select; change the delete to also require
last_heartbeat < cutoff so the removal only affects rows still stale (e.g., use
diesel::delete(workers::table.filter(workers::worker_id.eq_any(&dead_ids).and(workers::last_heartbeat.lt(cutoff))))
or equivalently include the last_heartbeat value with each selected id and
delete by matching both worker_id and that timestamp), keeping the symbols
workers::table, workers::worker_id, workers::last_heartbeat and the cutoff
variable referenced to ensure recovered workers are not deleted.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f27fe902-0f1f-4b66-8efd-b084664b8d84

📥 Commits

Reviewing files that changed from the base of the PR and between b20ad9c and 0162b5f.

📒 Files selected for processing (10)
  • crates/taskito-core/src/storage/diesel_common/archival.rs
  • crates/taskito-core/src/storage/diesel_common/dead_letter.rs
  • crates/taskito-core/src/storage/diesel_common/mod.rs
  • crates/taskito-core/src/storage/diesel_common/workers.rs
  • crates/taskito-core/src/storage/postgres/archival.rs
  • crates/taskito-core/src/storage/postgres/dead_letter.rs
  • crates/taskito-core/src/storage/postgres/workers.rs
  • crates/taskito-core/src/storage/sqlite/archival.rs
  • crates/taskito-core/src/storage/sqlite/dead_letter.rs
  • crates/taskito-core/src/storage/sqlite/workers.rs

Comment thread crates/taskito-core/src/storage/diesel_common/dead_letter.rs Outdated
Comment thread crates/taskito-core/src/storage/diesel_common/workers.rs Outdated
retry_dead previously masked every Diesel error as JobNotFound and did
not verify the dead-letter delete affected a row. Two concurrent retries
could both commit a fresh job for the same dead id. Match on
diesel::result::Error::NotFound explicitly, propagate other errors as
Storage, and assert the in-transaction delete removed exactly one row —
otherwise abort the transaction so the freshly inserted job rolls back.
reap_dead_workers scanned for stale workers, then deleted by worker_id
alone. A worker that heartbeated between the scan and the delete was
still erased, deregistering a healthy fleet member. Apply the
last_heartbeat < cutoff predicate to the DELETE as well so only workers
that are still stale at delete time are removed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant