From a7b1babf37462b06f08669be43db5ad73ce943c1 Mon Sep 17 00:00:00 2001 From: Bowen Li Date: Mon, 18 Oct 2021 20:43:50 +0800 Subject: [PATCH 01/10] add docs of broker entry metadata --- site2/docs/developing-binary-protocol.md | 58 ++++++++++++++---------- 1 file changed, 35 insertions(+), 23 deletions(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index 33861af0da7c9..60d290fbd28a6 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -39,32 +39,44 @@ Simple (payload-free) commands have this basic structure: Payload commands have this basic structure: -| Component | Description | Size (in bytes) | -|:-------------|:--------------------------------------------------------------------------------------------|:----------------| -| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 | -| commandSize | The size of the protobuf-serialized command | 4 | -| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | -| magicNumber | A 2-byte byte array (`0x0e01`) identifying the current format | 2 | -| checksum | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 | -| metadataSize | The size of the message [metadata](#message-metadata) | 4 | -| metadata | The message [metadata](#message-metadata) stored as a binary protobuf message | | -| payload | Anything left in the frame is considered the payload and can include any sequence of bytes | | +| Component | Necessity | Description | Size (in bytes) | +|:---------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------| +| totalSize | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 | +| commandSize | Required | The size of the protobuf-serialized command | 4 | +| message | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | +| magicNumberOfBrokerEntryMetadata | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata | 2 | +| brokerEntryMetadataSize | Optional | The size of the broker entry metadata | 4 | +| brokerEntryMetadata | Optional | The broker entry metadata stored as a binary protobuf message | | +| magicNumber | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 | +| checksum | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 | +| metadataSize | Required | The size of the message [metadata](#message-metadata) | 4 | +| metadata | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | | +| payload | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | | + +## Broker entry metadata +Broker entry metadata is stored alongside the message metadata as a serialized protobuf message. +It is created by the broker when the message arrived at the broker and passed without change to the consumer if configured. + +| Field | Necessity | Description | +|:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------| +| `broker_timestamp` | Optional | The timestamp when the message arrived at the broker (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | +| `index` | Optional | The index of the message, assigned by the broker ## Message metadata -Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed on unchanged to the consumer. - -| Field | Description | -|:-------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `producer_name` | The name of the producer that published the message | -| `sequence_id` | The sequence ID of the message, assigned by producer | -| `publish_time` | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | -| `properties` | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. | -| `replicated_from` *(optional)* | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published | -| `partition_key` *(optional)* | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose. Partition key is used as the message key. | -| `compression` *(optional)* | Signals that payload has been compressed and with which compression library | -| `uncompressed_size` *(optional)* | If compression is used, the producer must fill the uncompressed size field with the original payload size | -| `num_messages_in_batch` *(optional)* | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch | +Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without change to the consumer. + +| Field | Necessity | Description | +|:-------------------------|:----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `producer_name` | Required | The name of the producer that published the message | +| `sequence_id` | Required | The sequence ID of the message, assigned by producer | +| `publish_time` | Required | The publish timestamp in Unix time (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | +| `properties` | Required | A sequence of key/value pairs (using the [`KeyValue`](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L32) message). These are application-defined keys and values with no special meaning to Pulsar. | +| `replicated_from` | Optional | Indicates that the message has been replicated and specifies the name of the [cluster](reference-terminology.md#cluster) where the message was originally published | +| `partition_key` | Optional | While publishing on a partition topic, if the key is present, the hash of the key is used to determine which partition to choose. Partition key is used as the message key. | +| `compression` | Optional | Signals that payload has been compressed and with which compression library | +| `uncompressed_size` | Optional | If compression is used, the producer must fill the uncompressed size field with the original payload size | +| `num_messages_in_batch` | Optional | If this message is really a [batch](#batch-messages) of multiple entries, this field must be set to the number of messages in the batch | ### Batch messages From 70df1af05cd437396936c2f7c2e1be4d4f7e5098 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 10:56:20 +0800 Subject: [PATCH 02/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index 60d290fbd28a6..ca0bbe601502c 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -44,7 +44,7 @@ Payload commands have this basic structure: | totalSize | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 | | commandSize | Required | The size of the protobuf-serialized command | 4 | | message | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | -| magicNumberOfBrokerEntryMetadata | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata | 2 | +| magicNumberOfBrokerEntryMetadata | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata
**Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 | | brokerEntryMetadataSize | Optional | The size of the broker entry metadata | 4 | | brokerEntryMetadata | Optional | The broker entry metadata stored as a binary protobuf message | | | magicNumber | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 | From d4c95577afd8cbd771eb58ef32f332ac617536c9 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 10:56:42 +0800 Subject: [PATCH 03/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index ca0bbe601502c..baee5edc7d9ea 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -39,7 +39,7 @@ Simple (payload-free) commands have this basic structure: Payload commands have this basic structure: -| Component | Necessity | Description | Size (in bytes) | +| Component | Required or optional| Description | Size (in bytes) | |:---------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------| | totalSize | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 | | commandSize | Required | The size of the protobuf-serialized command | 4 | From 91ed135bcbebb8b3c0ca6a76e97d5937aa758463 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 10:56:52 +0800 Subject: [PATCH 04/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index baee5edc7d9ea..a51f6e40383bc 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -55,7 +55,7 @@ Payload commands have this basic structure: ## Broker entry metadata Broker entry metadata is stored alongside the message metadata as a serialized protobuf message. -It is created by the broker when the message arrived at the broker and passed without change to the consumer if configured. +It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured. | Field | Necessity | Description | |:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------| From 722aa42e376a5474dc7b294800bbcf67111f9369 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 10:56:59 +0800 Subject: [PATCH 05/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index a51f6e40383bc..fdd7c8909f7ba 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -57,7 +57,7 @@ Payload commands have this basic structure: Broker entry metadata is stored alongside the message metadata as a serialized protobuf message. It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured. -| Field | Necessity | Description | +| Field | Required or optional | Description | |:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------| | `broker_timestamp` | Optional | The timestamp when the message arrived at the broker (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | | `index` | Optional | The index of the message, assigned by the broker From f4b56218481e33f1604d249431bc9ef68d165904 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 11:00:14 +0800 Subject: [PATCH 06/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index fdd7c8909f7ba..77cc64012d69f 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -60,7 +60,7 @@ It is created by the broker when the message arrived at the broker and passed wi | Field | Required or optional | Description | |:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------| | `broker_timestamp` | Optional | The timestamp when the message arrived at the broker (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | -| `index` | Optional | The index of the message, assigned by the broker +| `index` | Optional | The index of the message. It is assigned by the broker. ## Message metadata From ea7df4bfc97e01570914aff413287e7b7af56d15 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 11:00:24 +0800 Subject: [PATCH 07/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index 77cc64012d69f..a9342acf9e0b7 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -64,7 +64,7 @@ It is created by the broker when the message arrived at the broker and passed wi ## Message metadata -Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without change to the consumer. +Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without changes to the consumer. | Field | Necessity | Description | |:-------------------------|:----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| From 5cadd08de1025216a8e4314efcaead1c6ed9f555 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 11:00:33 +0800 Subject: [PATCH 08/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index a9342acf9e0b7..c7ce46b45bcf6 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -66,7 +66,7 @@ It is created by the broker when the message arrived at the broker and passed wi Message metadata is stored alongside the application-specified payload as a serialized protobuf message. Metadata is created by the producer and passed without changes to the consumer. -| Field | Necessity | Description | +| Field | Required or optional | Description | |:-------------------------|:----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `producer_name` | Required | The name of the producer that published the message | | `sequence_id` | Required | The sequence ID of the message, assigned by producer | From 85eecd61197971ece97c74a561c576cd09a0bfbd Mon Sep 17 00:00:00 2001 From: Bowen Li Date: Tue, 19 Oct 2021 11:07:21 +0800 Subject: [PATCH 09/10] fix comments. --- site2/docs/developing-binary-protocol.md | 57 ++++++++++++------------ 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index c7ce46b45bcf6..8d2d855017647 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -29,31 +29,32 @@ The Pulsar protocol allows for two types of commands: Simple (payload-free) commands have this basic structure: -| Component | Description | Size (in bytes) | -|:------------|:----------------------------------------------------------------------------------------|:----------------| -| totalSize | The size of the frame, counting everything that comes after it (in bytes) | 4 | -| commandSize | The size of the protobuf-serialized command | 4 | -| message | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | +| Component | Description | Size (in bytes) | +|:--------------|:----------------------------------------------------------------------------------------|:----------------| +| `totalSize` | The size of the frame, counting everything that comes after it (in bytes) | 4 | +| `commandSize` | The size of the protobuf-serialized command | 4 | +| `message` | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | ### Payload commands Payload commands have this basic structure: -| Component | Required or optional| Description | Size (in bytes) | -|:---------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------| -| totalSize | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 | -| commandSize | Required | The size of the protobuf-serialized command | 4 | -| message | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | -| magicNumberOfBrokerEntryMetadata | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata
**Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 | -| brokerEntryMetadataSize | Optional | The size of the broker entry metadata | 4 | -| brokerEntryMetadata | Optional | The broker entry metadata stored as a binary protobuf message | | -| magicNumber | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 | -| checksum | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 | -| metadataSize | Required | The size of the message [metadata](#message-metadata) | 4 | -| metadata | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | | -| payload | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | | +| Component | Required or optional| Description | Size (in bytes) | +|:-----------------------------------|:----------|:--------------------------------------------------------------------------------------------|:----------------| +| `totalSize` | Required | The size of the frame, counting everything that comes after it (in bytes) | 4 | +| `commandSize` | Required | The size of the protobuf-serialized command | 4 | +| `message` | Required | The protobuf message serialized in a raw binary format (rather than in protobuf format) | | +| `magicNumberOfBrokerEntryMetadata` | Optional | A 2-byte byte array (`0x0e02`) identifying the broker entry metadata
**Note**: `magicNumberOfBrokerEntryMetadata` , `brokerEntryMetadataSize`, and `brokerEntryMetadata` should be used **together**. | 2 | +| `brokerEntryMetadataSize` | Optional | The size of the broker entry metadata | 4 | +| `brokerEntryMetadata` | Optional | The broker entry metadata stored as a binary protobuf message | | +| `magicNumber` | Required | A 2-byte byte array (`0x0e01`) identifying the current format | 2 | +| `checksum` | Required | A [CRC32-C checksum](http://www.evanjones.ca/crc32c.html) of everything that comes after it | 4 | +| `metadataSize` | Required | The size of the message [metadata](#message-metadata) | 4 | +| `metadata` | Required | The message [metadata](#message-metadata) stored as a binary protobuf message | | +| `payload` | Required | Anything left in the frame is considered the payload and can include any sequence of bytes | | ## Broker entry metadata + Broker entry metadata is stored alongside the message metadata as a serialized protobuf message. It is created by the broker when the message arrived at the broker and passed without changes to the consumer if configured. @@ -88,19 +89,19 @@ object. For a single batch, the payload format will look like this: -| Field | Description | -|:--------------|:------------------------------------------------------------| -| metadataSizeN | The size of the single message metadata serialized Protobuf | -| metadataN | Single message metadata | -| payloadN | Message payload passed by application | +| Field | Required or optional | Description | +|:----------------|:---------------------|:-----------------------------------------------------------| +| `metadataSizeN` | Required |The size of the single message metadata serialized Protobuf | +| `metadataN` | Required |Single message metadata | +| `payloadN` | Required |Message payload passed by application | Each metadata field looks like this; -| Field | Description | -|:---------------------------|:--------------------------------------------------------| -| properties | Application-defined properties | -| partition key *(optional)* | Key to indicate the hashing to a particular partition | -| payload_size | Size of the payload for the single message in the batch | +| Field | Required or optional | Description | +|:----------------|:----------------------|:--------------------------------------------------------| +| `properties` | Required | Application-defined properties | +| `partition key` | Optional | Key to indicate the hashing to a particular partition | +| `payload_size` | Required | Size of the payload for the single message in the batch | When compression is enabled, the whole batch will be compressed at once. From 42bc7555ab31df3bd0135a061e628bee8cf12d77 Mon Sep 17 00:00:00 2001 From: Bowen Li <27091925@qq.com> Date: Tue, 19 Oct 2021 11:44:46 +0800 Subject: [PATCH 10/10] Update site2/docs/developing-binary-protocol.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/developing-binary-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/developing-binary-protocol.md b/site2/docs/developing-binary-protocol.md index 8d2d855017647..56f7aef4d4059 100644 --- a/site2/docs/developing-binary-protocol.md +++ b/site2/docs/developing-binary-protocol.md @@ -60,7 +60,7 @@ It is created by the broker when the message arrived at the broker and passed wi | Field | Required or optional | Description | |:-------------------|:----------------|:------------------------------------------------------------------------------------------------------------------------------| -| `broker_timestamp` | Optional | The timestamp when the message arrived at the broker (i.e. as the number of milliseconds since January 1st, 1970 in UTC) | +| `broker_timestamp` | Optional | The timestamp when a message arrived at the broker (`id est` as the number of milliseconds since January 1st, 1970 in UTC) | | `index` | Optional | The index of the message. It is assigned by the broker. ## Message metadata