[improve][doc] Added docs for key shared subscription hashing schemes by asafm · Pull Request #18878 · apache/pulsar

asafm · 2022-12-11T15:12:04Z

Motivation

Key Shared subscription type has multiple algorithms for mapping a message key to a consumer, yet they are not documented, leaving the user with no easy way of knowing how a Key Shared Subscription works and what are the possible options for configuration.

Modifications

Added documentation under Messaging / Concepts / Subscriptions / Key Shared

Verifying this change

Make sure that the change passes the CI checks.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: asafm#1

…d explains what happens exactly when a consumer connects and disconnects.

tisonkun

Thanks for your contribution @asafm!

I give a rough pass and comments inline.

tisonkun · 2022-12-12T01:30:28Z

+There are three types of mapping algorithms dictating how to select a consumer for a given message key (or ordering key): Sticky, Auto-split Hash Range, and Auto-split Consistent Hashing. Before using the algorithm, the message key (or ordering key) is first passed to a hash function (e.g., Murmur3 32-bit), yielding a 32-bit integer hash. That hash number is then fed to the algorithm to select a consumer from the existing connected consumers.
+When a new consumer is connected and thus added to the list of connected consumers, the algorithm re-adjusts the mapping such that some keys currently mapped to existing consumers will be mapped to the newly added consumer. When a consumer is disconnected, thus removed from the list of connected consumers, keys mapped to it will be mapped to other consumers. The sections below will explain how a consumer is selected given the message hash and how the mapping is adjusted given a new consumer is connected or an existing consumer disconnects for each algorithm.


If you intend to write these two lines in different paragraphs, insert a blank line between them. Otherwise, this paragraph reads too long at first glance.

I will add a new line.

tisonkun · 2022-12-12T01:32:16Z

+##### Auto-split Hash Range
+The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received ( 0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.


We may try to insert a blank line between elements even when it's not required. Perhaps we need to write it explicitly in the documentation writing style guide. So it's now not a blocker.

But I read you also insert blank lines between elements below, then let's keep the style consistent.

Suggested change

##### Auto-split Hash Range

The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received ( 0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.

##### Auto-split Hash Range

The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received (0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.

I'll add a new line in all headings I've added to be consistent.

tisonkun · 2022-12-12T01:32:26Z

+Suppose we have 4 consumers (C1, C2, C3 and C4), then:
+```


ditto these cases

Added new lines everywhere :)

Anonymitaet · 2022-12-12T03:52:25Z

FYI: markdown doc changes can skip this step

pulsar/.github/changes-filter.yaml

Line 9 in 346b04a

- '*.md'

Anonymitaet · 2022-12-12T03:55:53Z

+There are three types of mapping algorithms dictating how to select a consumer for a given message key (or ordering key): Sticky, Auto-split Hash Range, and Auto-split Consistent Hashing. Before using the algorithm, the message key (or ordering key) is first passed to a hash function (e.g., Murmur3 32-bit), yielding a 32-bit integer hash. That hash number is then fed to the algorithm to select a consumer from the existing connected consumers.
+When a new consumer is connected and thus added to the list of connected consumers, the algorithm re-adjusts the mapping such that some keys currently mapped to existing consumers will be mapped to the newly added consumer. When a consumer is disconnected, thus removed from the list of connected consumers, keys mapped to it will be mapped to other consumers. The sections below will explain how a consumer is selected given the message hash and how the mapping is adjusted given a new consumer is connected or an existing consumer disconnects for each algorithm.


Suggest changing the chunks to ordered steps and using workflow (illustrate) to explain the process and concept more clearly. For example, https://pulsar.apache.org/docs/next/schema-overview/#producer-side

I converted it into a list of steps (2) and added an ASCII diagram for it.

asafm · 2022-12-12T10:16:57Z

FYI: markdown doc changes can skip this step

pulsar/.github/changes-filter.yaml

Line 9 in 346b04a

- '*.md'

@Anonymitaet Maybe we should note that in the documentation contribution guide.
I followed https://pulsar.apache.org/contribute/document-intro/, which says:

and here https://pulsar.apache.org/contribute/develop-labels/ it says

Anonymitaet · 2022-12-13T13:06:53Z

Maybe we should note that in the documentation contribution guide.

Yes, good catch, I've added a tip in apache/pulsar-site#327, PTAL

asafm · 2022-12-13T13:29:38Z

 #### Key_Shared

-In the *Key_Shared* type, multiple consumers can attach to the same subscription. Messages are delivered in distribution across consumers and messages with the same key or same ordering key are delivered to only one consumer. No matter how many times the message is re-delivered, it is delivered to the same consumer. When a consumer connects or disconnects, it causes the served consumer to change some message keys.
+In the *Key_Shared* type, multiple consumers can attach to the same subscription. Messages are delivered in distribution across consumers and messages with the same key or same ordering key are delivered to only one consumer. No matter how many times the message is re-delivered, it is delivered to the same consumer. 


asafm · 2022-12-13T13:30:13Z

+
+The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received ( 0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.
+
+Example:


asafm · 2022-12-13T13:30:54Z

+
+When a consumer is disconnected its region will be merged into the region on its right. Examples:
+
+C4 is disconnected:


asafm · 2022-12-13T13:31:43Z

+       ' - , _ _ _ ,  '
+```
+
+When adding a consumer, we mark 100 points on that circle and associate them to the newly added consumer. For each number between 1 and 100, we concatenate the consumer name to that number and run the hash function on it to get the location of the point on the circle that will be marked. For Example, if the consumer name is "orders-aggregator-pod-2345-consumer" then we would mark 100 points on that circle:


asafm · 2022-12-13T13:32:17Z

+
+When a consumer is added, we add 100 marking points to the circle as explained before. Due to the uniform distribution of the hash function, those 100 points act as if the new consumer takes a small slice of keys out of each existing consumer. It maintains the even distribution, on the trade-off that it impacts all existing consumers. [This video](https://www.youtube.com/watch?v=zaRkONvyGr8) explains the concept of Consistent Hashing quite well (the only difference is that in Pulsar's case we used K points instead of K hash functions as noted in the comments)
+
+##### Sticky


asafm · 2022-12-13T13:32:52Z

+
+Consistent Hashing will be used instead of Hash Range for Auto-split if the broker configuration `subscriptionKeySharedUseConsistentHashing` is enabled.
+
+##### Preserving order of processing


asafm · 2022-12-13T15:25:18Z

What is left is that I need to know how to make the diagrams consistent with the rest of the documentation

tisonkun · 2022-12-13T15:29:31Z

@asafm Yes. It seems authors write diagrams with their preferred tools. For the latest diagrams, you can contact with @DaveDuggins to see what's his tool.

And we may later find a best practice to document on the contribution guide documentation section.

Anonymitaet · 2022-12-14T07:06:40Z

What is left is that I need to know how to make the diagrams consistent with the rest of the documentation

Normally, we use LucidChart to draw illustrations, e.g., this is what I'm working on: https://lucid.app/lucidchart/323370a9-0466-49e1-836a-968885aaf23a/edit?invitationId=inv_55ab3a55-7f98-4943-baa4-a8be8ddf7fac&page=0_0#

tisonkun · 2022-12-14T07:08:33Z

What is left is that I need to know how to make the diagrams consistent with the rest of the documentation

Normally, we use LucidChart to draw illustrations, e.g., this is what I'm working on: https://lucid.app/lucidchart/323370a9-0466-49e1-836a-968885aaf23a/edit?invitationId=inv_55ab3a55-7f98-4943-baa4-a8be8ddf7fac&page=0_0#

@asafm as the PR merged now, you may send a new patch to update the diagrams :)

asafm · 2022-12-14T16:23:26Z

@tisonkun Thanks :) Better that than no explanation on shared subscription hashing, for sure.

…apache#18878)

asafm added 2 commits December 8, 2022 18:20

First draft

88c70b1

Adding explanation on hashing schemes for key shared subscriptions an…

96cc85e

…d explains what happens exactly when a consumer connects and disconnects.

github-actions Bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Dec 11, 2022

tisonkun requested review from codelipenghui, momo-jun and tisonkun December 12, 2022 01:28

tisonkun reviewed Dec 12, 2022

View reviewed changes

Anonymitaet reviewed Dec 12, 2022

View reviewed changes

PR Fixes #1

3ad593f

asafm commented Dec 13, 2022

View reviewed changes

This was referenced Dec 13, 2022

[improve][doc] add a tip for skipping test procedure for md contribution apache/pulsar-site#327

Closed

[fix] ready-to-test is not applicible for docs only changes apache/pulsar-site#328

Merged

tisonkun approved these changes Dec 13, 2022

View reviewed changes

Anonymitaet approved these changes Dec 14, 2022

View reviewed changes

Anonymitaet added this to the 2.11.0 milestone Dec 14, 2022

Anonymitaet merged commit 52c382a into apache:master Dec 14, 2022

lifepuzzlefun pushed a commit to lifepuzzlefun/pulsar that referenced this pull request Jan 10, 2023

[improve][doc] Added docs for key shared subscription hashing schemes (…

e5a3fc0

…apache#18878)

		There are three types of mapping algorithms dictating how to select a consumer for a given message key (or ordering key): Sticky, Auto-split Hash Range, and Auto-split Consistent Hashing. Before using the algorithm, the message key (or ordering key) is first passed to a hash function (e.g., Murmur3 32-bit), yielding a 32-bit integer hash. That hash number is then fed to the algorithm to select a consumer from the existing connected consumers.
		When a new consumer is connected and thus added to the list of connected consumers, the algorithm re-adjusts the mapping such that some keys currently mapped to existing consumers will be mapped to the newly added consumer. When a consumer is disconnected, thus removed from the list of connected consumers, keys mapped to it will be mapped to other consumers. The sections below will explain how a consumer is selected given the message hash and how the mapping is adjusted given a new consumer is connected or an existing consumer disconnects for each algorithm.

		##### Auto-split Hash Range
		The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received ( 0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.


		The algorithm assumes there is a range of numbers between 0 to 2^16 (65,536). Each consumer is mapped into a single region in this range, so all mapped regions cover the entire range, and no regions overlap. A consumer is selected for a given key by running a modulo operation on the message hash by the range size (65,536). The number received ( 0 <= i < 65,536) is contained within a single region. The consumer mapped to that region is the one selected.

		Example:


		When a consumer is disconnected its region will be merged into the region on its right. Examples:

		C4 is disconnected:


		When a consumer is added, we add 100 marking points to the circle as explained before. Due to the uniform distribution of the hash function, those 100 points act as if the new consumer takes a small slice of keys out of each existing consumer. It maintains the even distribution, on the trade-off that it impacts all existing consumers. [This video](https://www.youtube.com/watch?v=zaRkONvyGr8) explains the concept of Consistent Hashing quite well (the only difference is that in Pulsar's case we used K points instead of K hash functions as noted in the comments)

		##### Sticky


		Consistent Hashing will be used instead of Hash Range for Auto-split if the broker configuration `subscriptionKeySharedUseConsistentHashing` is enabled.

		##### Preserving order of processing

Uh oh!

Conversation

asafm commented Dec 11, 2022

Motivation

Modifications

Verifying this change

Documentation

Matching PR in forked repository

Uh oh!

tisonkun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anonymitaet commented Dec 12, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asafm commented Dec 12, 2022

Uh oh!

Anonymitaet commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asafm commented Dec 13, 2022

Uh oh!

tisonkun commented Dec 13, 2022

Uh oh!

Anonymitaet commented Dec 14, 2022

Uh oh!

tisonkun commented Dec 14, 2022

Uh oh!

asafm commented Dec 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Anonymitaet commented Dec 13, 2022 •

edited

Loading