Skip to content

[docs] Improve documentation about batching #5989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sijie merged 2 commits into from
Jan 23, 2020
Merged

[docs] Improve documentation about batching #5989

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
15492d7
Improve documentation about batching
gmethvin Jan 3, 2020
842a7e5
Merge remote-tracking branch 'apache/master' into batching-docs
sijie Jan 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions site2/docs/concepts-messaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,14 @@ Messages published by producers can be compressed during transportation in order

### Batching

If batching is enabled, the producer will accumulate and send a batch of messages in a single request. Batching size is defined by the maximum number of messages and maximum publish latency.
If batching is enabled, the producer will accumulate and send a batch of messages in a single request. Batch size is defined by the maximum number of messages and maximum publish latency.

Batches are tracked and stored by Pulsar as batches rather than as individual messages. Under the hood the consumer unbundles these batches into individual messages. Since the messages are stored as batches, the backlog size will also represent the total number of batches rather than the total number of messages.

Scheduled messages (using `deliverAt` or `deliverAfter`) are always sent as individual messages even when batching is enabled.

> Note
> Since batches are tracked as single units, a batch will only be considered acknowledged when all its messages are acknowledged by the consumer. This means unexpected failures, negative acknowledgements, and acknowledgement timeouts can result in redelivery of all messages in the batch, even if some of the messages have already been acknowledged.
Comment on lines +58 to +59

@Lanayx Lanayx Jan 4, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just noting is not enough, it should be pointed out that this is a buggy behavior and should be fixed in future releases.

@cdbartholomew cdbartholomew Jan 4, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #5990 for a fix for this from @zzzming.

@gmethvin gmethvin Jan 5, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That fix is only on the client side, so batches may still be fully redelivered if the consumer crashes or restarts, though.

@cdbartholomew cdbartholomew Jan 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. That fix improves the situation, but doesn't resolve it entirely.

@gmethvin gmethvin Jan 7, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we agree it's definitely a bug, then let's get the fix and this documentation update in the next release, and I'll remove the part about negative acks.

@sijie sijie Jan 7, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This is a bug and fixes at the client side are not enough. Just fixing this at the client side is actually making the behavior much more confused. We need to get this documentation merged first and then fix the behavior fundamentally in the next release.


## Consumers

Expand Down Expand Up @@ -89,12 +96,18 @@ In the exclusive and failover subscription modes, consumers only negatively ackn

In the shared and Key_Shared subscription modes, you can negatively acknowledge messages individually.

> Note
> If batching is enabled, other messages in the same batch may be redelivered to the consumer as well as the negatively acknowledged messages.

### Acknowledgement timeout

When a message is not consumed successfully, and you want to trigger the broker to redeliver the message automatically, you can adopt the unacknowledged message automatic re-delivery mechanism. Client will track the unacknowledged messages within the entire `acktimeout` time range, and send a `redeliver unacknowledged messages` request to the broker automatically when the acknowledgement timeout is specified.

> Note
> If batching is enabled, other messages in the same batch may be redelivered to the consumer as well as the unacknowledged messages.

> Note
> Use negative acknowledgement prior to acknowledgement timeout. Negative acknowledgement controls re-delivery of individual messages with more precise, and avoids invalid redeliveries when the message processing time exceeds the acknowledgement timeout.
> Prefer negative acknowledgements over acknowledgement timeout. Negative acknowledgement controls the re-delivery of individual messages with more precision, and avoids invalid redeliveries when the message processing time exceeds the acknowledgement timeout.

### Dead letter topic

Expand Down