feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise)#684
Open
dgafka wants to merge 32 commits into
Open
feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise)#684dgafka wants to merge 32 commits into
dgafka wants to merge 32 commits into
Conversation
…a TCP_NODELAY default
…eue-full backpressure
…mmand before commit
…en by command handler batch publishing tests with delay and ttl across providers
…cycle - catch \RdKafka\Exception instead of the undefined KafkaException so producer queue-full backpressure actually polls and retries rather than failing the publish on the first full-queue signal - require AsyncPublishingRegistry in the SQS and Enqueue outbound adapters, matching DBAL and Redis, removing the nullable dereference - drop the register_shutdown_function auto-flush; unresolved deliveries are now flushed via explicit flushUnawaitedDeliveries and the backlog limit
…nd error channel routing, shutdown flush of unawaited deliveries
…t package testing passes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this change proposed?
Publishing to a message broker blocks the handler on one full round-trip per message — a
handler emitting many messages pays N sequential broker round-trips, which caps publishing
throughput as systems scale.
Ecotone now provides an Enterprise feature that makes publishing scale, while still keeping
all the delivery guarantees end users rely on: every message is still confirmed by the
broker, a failed delivery still fails the business operation or is routed to the error
channel, and every failure points to the exact message that failed — nothing is ever lost
silently.
This is achieved by firing messages to the broker immediately, batching them natively per
provider, and deferring confirmation awaiting to the last responsible moment — messages
travel to the broker while the handler keeps working, and all confirmations are collected
before the operation completes.
Benchmarks
Publishing 1,000 messages, relative to each provider's previous message-by-message
synchronous publishing (= 100%). Mode of 10 iterations, all delivery confirmations awaited:
Batching + async sending publishes the workload as provider-native batches with
confirmation awaiting deferred; async sending fires individual messages without waiting.
DBAL and Redis write synchronously, so batching is their applicable feature.
Resulting flow
graph TD A[Handler emits messages] --> B[Collector gathers them] B -->|one BatchMessage| C[Adapter: provider-native batch publish, no wait] C --> D[Registry: PendingDelivery per publish] D --> E[Waiter interceptor: await all confirmations] E -->|all confirmed| F[Operation completes] E -->|some failed| G[Only failed messages to error channel, or operation fails]Enabling
Example
Pull Request Contribution Terms