feat: first-class topic registry#462
Merged
Merged
Conversation
Declared log topics retain publishes with zero subscribers, with an optional per-topic retention window; removes the S28 late-join boundary.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughAdds a declared-topic registry across storage backends, retains log publishes for declared topics without subscribers, and exposes declaration/listing APIs through Java, Node, and Python bindings with tests and documentation. ChangesDeclared topic registry
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant SDK
participant NativeBindings
participant Storage
participant TopicRegistry
participant TopicMessages
SDK->>NativeBindings: declare topic
NativeBindings->>Storage: declare_topic(name, mode, retention_ms)
Storage->>TopicRegistry: upsert topic metadata
SDK->>NativeBindings: publish before subscription
NativeBindings->>Storage: publish_to_topic
Storage->>TopicRegistry: get_topic
Storage->>TopicMessages: store retained message
SDK->>NativeBindings: list declared topics
NativeBindings->>Storage: list_declared_topics
Storage-->>SDK: topic metadata
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/taskito-core/src/pubsub.rs`:
- Around line 475-494: Update
declare_topic_is_idempotent_and_preserves_created_at so the first and second
declare_topic calls occur in distinct milliseconds, waiting or otherwise
advancing the clock between them before asserting second.created_at equals
first.created_at.
In `@crates/taskito-core/src/storage/traits.rs`:
- Around line 293-300: Define the declare_topic contract in
crates/taskito-core/src/storage/traits.rs to accept only mode "log" and
non-negative retention_ms values. Enforce these invariants before persistence in
declare_topic for crates/taskito-core/src/storage/postgres/pubsub.rs (lines
14-30), crates/taskito-core/src/storage/sqlite/pubsub.rs (lines 14-30), and
crates/taskito-core/src/storage/redis_backend/pubsub.rs (lines 794-812),
rejecting invalid inputs before the upsert or hash write.
In `@docs/content/docs/shared/guides/core/pubsub.mdx`:
- Around line 501-506: Update the “Declared topics” explanation to qualify
late-join behavior by retention: declared topics retain publishes without an
existing durable log subscription, but finite retention may expire messages
before a later subscriber arrives, so the whole backlog is not guaranteed.
Replace “live log subscriber” terminology with explicit references to an
existing durable log subscription, independent of whether its consumer process
is online, and apply the same clarification to the corresponding text later in
the guide.
In `@sdks/java/src/main/java/org/byteveda/taskito/DefaultTaskito.java`:
- Around line 934-936: Reject negative retention values before declaring topics:
in sdks/java/src/main/java/org/byteveda/taskito/DefaultTaskito.java lines
934-936, validate the Duration in declareTopic before converting it to
milliseconds; in sdks/node/src/queue.ts lines 661-666, validate retention before
rounding and forwarding it. Preserve null/omitted retention behavior and use the
SDKs’ existing argument-validation error convention.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 535a4547-d257-44d8-ab33-c9e49001c63e
📒 Files selected for processing (37)
crates/taskito-core/BINDING_CONTRACT.mdcrates/taskito-core/migrations/m0006_topics.rscrates/taskito-core/src/pubsub.rscrates/taskito-core/src/storage/diesel_common/pubsub.rscrates/taskito-core/src/storage/mod.rscrates/taskito-core/src/storage/models.rscrates/taskito-core/src/storage/postgres/pubsub.rscrates/taskito-core/src/storage/records.rscrates/taskito-core/src/storage/redis_backend/pubsub.rscrates/taskito-core/src/storage/schema.rscrates/taskito-core/src/storage/sqlite/pubsub.rscrates/taskito-core/src/storage/traits.rscrates/taskito-core/tests/rust/storage_tests.rscrates/taskito-java/src/convert.rscrates/taskito-java/src/queue/pubsub.rscrates/taskito-node/src/convert/mod.rscrates/taskito-node/src/convert/pubsub.rscrates/taskito-node/src/queue/pubsub.rscrates/taskito-python/src/py_queue/pubsub.rsdocs/content/docs/java/api-reference/queue/pubsub.mdxdocs/content/docs/node/api-reference/queue/pubsub.mdxdocs/content/docs/python/api-reference/queue/pubsub.mdxdocs/content/docs/shared/guides/core/pubsub.mdxsdks/java/src/main/java/org/byteveda/taskito/DefaultTaskito.javasdks/java/src/main/java/org/byteveda/taskito/Taskito.javasdks/java/src/main/java/org/byteveda/taskito/internal/JniQueueBackend.javasdks/java/src/main/java/org/byteveda/taskito/internal/NativeQueue.javasdks/java/src/main/java/org/byteveda/taskito/model/Topic.javasdks/java/src/main/java/org/byteveda/taskito/spi/QueueBackend.javasdks/java/src/test/java/org/byteveda/taskito/core/TopicRegistryTest.javasdks/node/src/native.tssdks/node/src/queue.tssdks/node/src/types.tssdks/node/test/core/topics-registry.test.tssdks/python/taskito/_taskito.pyisdks/python/taskito/mixins/pubsub.pysdks/python/tests/core/test_pubsub_log.py
# Conflicts: # docs/content/docs/java/api-reference/queue/pubsub.mdx # docs/content/docs/node/api-reference/queue/pubsub.mdx # docs/content/docs/python/api-reference/queue/pubsub.mdx # sdks/java/src/main/java/org/byteveda/taskito/DefaultTaskito.java # sdks/java/src/main/java/org/byteveda/taskito/Taskito.java # sdks/python/taskito/mixins/pubsub.py
declare_topic now rejects a non-"log" mode and a negative retention_ms (which would expire messages immediately or overflow now+retention) in every backend; the created_at test sleeps so preservation is observable.
Collaborator
Author
|
Addressed the CodeRabbit review (commits
|
The cold macOS/Windows smoke runners starved on five per-file worker startups (a 40s waitFor still timed out). Four consumers now share one worker, so once it's up they all drain — one startup instead of four.
Same cold-runner robustness as the Node suite: four consumers share one worker instead of one worker lifecycle per test.
Consolidate the three managed-consumer cases into one worker so the cold-compile Java smoke jobs don't starve on repeated worker startups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a first-class topic registry — the second S28 log+cursor pub/sub follow-up. Declaring a topic lets its log messages be retained even with zero subscribers at publish time, removing the late-join boundary, with an optional per-topic retention window.
Core
m0006—topicstable (namePK,modedefault'log',retention_ms,created_at).Storage:declare_topic(idempotent upsert),get_topic,list_declared_topics— SQLite, Postgres, Redis (atopicshash).publish_to_topicwrites a log row when the topic is a declared log topic or alogsubscriber exists (was: only the latter). The registry lookup runs only when there is no log subscriber, so the hot path is unchanged. A declared topic withretention_msstampsexpires_at = now + retention_mson the message, so the existing expiry sweep reclaims sub-less messages — no sweep change.Cross-SDK
queue.declare_topic(name, *, retention=<seconds>)/list_declared_topics()queue.declareTopic(name, { retention })/listDeclaredTopics()taskito.declareTopic(name[, Duration retention])/listDeclaredTopics()Tests
test_topic_registry): declare → publish with zero subs → latesubscribe_logreads it; retention round-trips; idempotent re-declare preservescreated_at. SQLite green; Postgres/Redis run in CI.TopicRegistryTest, 2) declare-then-late-join round-trips. Full suites green.BINDING_CONTRACT.md+ shared guide "Declared topics" section + 3 API refs.Summary by CodeRabbit
New Features
Documentation
Tests