[Issue 5562][docs]Modify schema compatibility check doc by congbobo184 · Pull Request #5757 · apache/pulsar

congbobo184 · 2019-11-27T12:21:21Z

Fixes #5562
In order to add the doc for #5227 , to solve the issue #5562

Motivation

modify the doc.

This reverts commit cdf7735.

congbobo184 · 2019-11-27T12:23:59Z

@jennifer88huang @Anonymitaet PTAL!

Anonymitaet · 2019-11-27T12:33:41Z

@congbobo184 I'll review later.

congbobo184 · 2019-11-28T06:13:46Z

run cpp tests
run java8 tests

Anonymitaet · 2019-11-29T02:05:45Z

+
+<td> 
+
+The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.


Suggested change

The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.

* If you upgrade a lower version to Pulsar 2.5.0, then it uses the `UNDEFINED` strategy, at this time, it actually uses the compatibility strategy of the topic.

* If you use Pulsar 2.5.0 and a higher version, then it uses the `UNDEFINED` strategy, at this time, it actually uses the `FULL` compatibility strategy.

Anonymitaet · 2019-11-29T02:07:42Z

+
+<td> 
+
+NONE


Suggested change

NONE

It depends

Anonymitaet · 2019-11-29T02:07:48Z

+
+NONE
+
+</td> 


Suggested change

</td>

It depends

Anonymitaet · 2019-11-29T02:07:58Z

+
+<td> 
+
+NONE


Suggested change

NONE

It depends

Anonymitaet · 2019-11-29T02:10:51Z

+
+<th>
+
+Note 


Suggested change

Note

Description

Anonymitaet · 2019-11-29T02:11:37Z

+
+<td> 
+
+The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.


Suggested change

The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.

* If you upgrade a lower version to Pulsar 2.5.0, then it uses the `UNDEFINED` strategy, at this time, it actually uses the compatibility strategy of the topic.

* If you use Pulsar 2.5.0 and a higher version, then it uses the `UNDEFINED` strategy, at this time, it actually uses the `FULL` compatibility strategy.

Anonymitaet · 2019-11-29T02:46:22Z


 </table>
+
+## Schema verification for producers


I've moved this part before order of upgrading clients, please delete lines 857-1017.

Anonymitaet · 2019-11-29T02:46:48Z

+</tr>
+
+</table>
+


Add schema verification here. I've re-written some sentences as well.

Suggested change

## Schema verification

When a producer or a consumer tries to connect to a topic, a broker performs some checks to verify a schema.

### Producer

When a producer tries to connect to a topic (suppose ignore the schema auto creation), a broker does the following checks:

* Check if the schema carried by the producer exists in the schema registry or not.

* If the schema is already registered, then the producer is connected to a broker and produce messages with that schema.

* If the schema is not registered, then Pulsar verifies if the schema is allowed to be registered based on the configured compatibility check strategy.

### Consumer

When a consumer tries to connect to a topic, a broker checks if a carried schema is compatible with a registered schema based on the configured schema compatibility check strategy.

<table style="table">

<tr>

<th>

Compatibility check strategy

</th>

<th>

Check logic

</th>

</tr>

<tr>

<td>

`ALWAYS_COMPATIBLE`

</td>

<td>

All pass

</td>

</tr>

<tr>

<td>

`ALWAYS_INCOMPATIBLE`

</td>

<td>

No pass

</td>

</tr>

<tr>

<td>

`BACKWARD`

</td>

<td>

Can read the last schema

</td>

</tr>

<tr>

<td>

`BACKWARD_TRANSITIVE`

</td>

<td>

Can read all schemas

</td>

</tr>

<tr>

<td>

`FORWARD`

</td>

<td>

Can read the last schema

</td>

</tr>

<tr>

<td>

`FORWARD_TRANSITIVE`

</td>

<td>

Can read the last schema

</td>

</tr>

<tr>

<td>

`FULL`

</td>

<td>

Can read the last schema

</td>

</tr>

<tr>

<td>

`FULL_TRANSITIVE`

</td>

<td>

Can read all schemas

</td>

</tr>

</table>

congbobo184 · 2019-11-29T03:23:44Z

@sijie

bin/pulsar-admin namespaces set-schema-autoupdate-strategy --compatibility <compatibility-level> tenant/namespace
bin/pulsar-admin namespaces get-schema-autoupdate-strategy --compatibility <compatibility-level> tenant/namespace

I think this two command should not be provided to users.
It's easy to confuse users following commands :

bin/pulsar-admin namespaces set-schema-compatibility-strategy --compatibility <compatibility-level> tenant/namespace
bin/pulsar-admin namespaces get-schema-compatibility-strategy --compatibility <compatibility-level> tenant/namespace
bin/pulsar-admin namespaces get-is-allow-auto-update-schema --disable tenant/namespace

sijie · 2019-11-29T03:30:25Z

@congbobo184 if those two commands are deprecated, you can add hidden flag to hide them.

congbobo184 · 2019-11-29T03:36:58Z

This seems like a good way.

Anonymitaet · 2019-11-29T02:53:47Z

-    * If it is a new schema and it passes the compatibility check, the broker registers a new schema automatically for the topic.
-
-    * If the schema does not pass the compatibility check, the broker does not register a schema.
+    * If the schema is registered, producer is connected. 


Suggested change

* If the schema is registered, producer is connected.

* If the schema is registered, a producer is connected to a broker.

Anonymitaet · 2019-11-29T02:54:03Z

-    * If the schema does not pass the compatibility check, the broker does not register a schema.
+    * If the schema is registered, producer is connected. 
+
+    * If the schema isn't registered


Suggested change

* If the schema isn't registered

* If the schema is not registered:

Anonymitaet · 2019-11-29T02:54:26Z

+
+    * If the schema isn't registered
+
+        * If`isAllowAutoUpdateSchema` = false, producer is rejected.


Suggested change

* If`isAllowAutoUpdateSchema` = false, producer is rejected.

* If `isAllowAutoUpdateSchema` sets to **false**, the producer is rejected to connect to a broker.

Anonymitaet · 2019-11-29T02:54:43Z

+
+        * If`isAllowAutoUpdateSchema` = false, producer is rejected.
+
+        * If `isAllowAutoUpdateSchema` = true


Suggested change

* If `isAllowAutoUpdateSchema` = true

* If `isAllowAutoUpdateSchema` sets to **true**:

Anonymitaet · 2019-11-29T02:57:39Z

+
+        * If `isAllowAutoUpdateSchema` = true
+
+            * If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.


Suggested change

* If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.

* If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.

Suggested change

* If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.

* If the schema passes the compatibility check, then the broker registers a new schema automatically for the topic and the producer is connected.

Anonymitaet · 2019-11-29T03:05:23Z

-    * If the **schema does not pass the compatibility check**, the consumer is rejected and disconnected.
-
-![AutoUpdate Producer](assets/schema-autoupdate-consumer.png)
+        * If schema pass the compatibility check, the consumer is connected.


Suggested change

* If schema pass the compatibility check, the consumer is connected.

* If the schema passes the compatibility check, then the consumer is connected to the broker.

Anonymitaet · 2019-11-29T03:05:56Z

-![AutoUpdate Producer](assets/schema-autoupdate-consumer.png)
+        * If schema pass the compatibility check, the consumer is connected.
+
+        * If schema does not pass the compatibility, the consumer is rejected.


Suggested change

* If schema does not pass the compatibility, the consumer is rejected.

* If the schema does not pass the compatibility check, then the consumer is rejected to connect to the broker.

Anonymitaet · 2019-11-29T03:43:09Z


 ```bash
-bin/pulsar-admin namespaces set-schema-autoupdate-strategy --disabled tenant/namespace
+bin/pulsar-admin namespaces set-is-allow-auto-update-schema --disabled tenant/namespace


Do we consider adding enable AutoUpdate before this part?

From line 82:

Suggested change

bin/pulsar-admin namespaces set-is-allow-auto-update-schema --disabled tenant/namespace

* [Enable AutoUpdate](#enable-autoupdate)

* [Disable AutoUpdate](#disable-autoupdate)

* [Adjust compatibility](#adjust-compatibility)

#### Enable AutoUpdate

To enable `AutoUpdate` on a namespace, you can use the `pulsar-admin` command.

```bash

bin/pulsar-admin namespaces set-is-allow-auto-update-schema --enable tenant/namespace

Anonymitaet · 2019-11-29T03:51:25Z

+5. If no, the broker judge whether this namespace can automatically create schema.
+   isAllowAutoUpdateSchema can be modified by Admin-Api and REST-API 


Suggested change

5. If no, the broker judge whether this namespace can automatically create schema.

isAllowAutoUpdateSchema can be modified by Admin-Api and REST-API

5. If no, the broker verifies whether a schema can be automatically created in this namespace:

* If `isAllowAutoUpdateSchema` sets to **true**, then a schema can be created, and the broker validates the schema based on the schema compatibility check strategy defined for the topic.

* If `isAllowAutoUpdateSchema` sets to **false**, then a schema can not be created, and the consumer is rejected to connect to the broker.

**Tip**:

`isAllowAutoUpdateSchema` can be set via **Pulsar admin API** or **REST API.**

For how to set `isAllowAutoUpdateSchema` via Pulsar admin API, see [Manage AutoUpdate Strategy](schema-manage.md/#manage-autoupdate-strategy).

Anonymitaet · 2019-11-29T03:59:20Z

+   isAllowAutoUpdateSchema can be modified by Admin-Api and REST-API 

-5. If no, the broker validates the schema based on the schema compatibility check strategy defined for the topic. 
+6. If yes, the broker validates the schema based on the schema compatibility check strategy defined for the topic. 


I've moved the line 546 to the step 5.
If it is correct, can we delete the line 546?

yes, we can delete the line 546 👍 :)

Anonymitaet · 2019-11-29T04:03:09Z

-5. If no, the broker validates the schema based on the schema compatibility check strategy defined for the topic. 
-
-6. If the schema is compatible, the broker stores it and returns the schema version to the consumer. 
+3. If the schema is compatible, the consumer will be connected. 


Suggested change

3. If the schema is compatible, the consumer will be connected.

3. If the schema is compatible, the consumer is connected to the broker.

Anonymitaet · 2019-11-29T04:03:27Z

+3. If the schema is compatible, the consumer will be connected. 

-7. If the schema is incompatible, the consumer will be disconnected.
+4. If the schema is incompatible, the consumer will be disconnected.


Suggested change

4. If the schema is incompatible, the consumer will be disconnected.

4. If the schema is incompatible, the consumer is disconnected to the broker.

Anonymitaet · 2019-11-29T04:04:06Z

@congbobo184 as we discussed before, we finalize contents first and then re-draw the workflow, so pls do not add images now.

congbobo184 · 2019-11-29T06:34:51Z

@sijie where does the flag add? pulsar-client-tools or broker reject this request? if it is in pular-client-tools, how to add it?

sijie · 2019-11-29T08:29:10Z

@congbobo184 this is one of the examples: https://github.com/apache/pulsar/blob/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java#L67

congbobo184 · 2019-11-29T08:33:43Z

Thanks, i will look carefully and add hidden flag to hide them. :)

Anonymitaet · 2019-11-29T10:35:07Z

 ## Schema compatibility check strategy

-Pulsar has 8 schema compatibility check strategies, which are summarized in the following table.
+Pulsar has 9 schema compatibility check strategies, which are summarized in the following table.


Suggested change

Pulsar has 9 schema compatibility check strategies, which are summarized in the following table.

Pulsar has 8 schema compatibility check strategies, which are summarized in the following table.

Anonymitaet · 2019-11-29T11:24:25Z

+* [Enable AutoUpdate](#enable-autoupdate)
+
 * [Disable AutoUpdate](#disable-autoupdate)


As we discussed just now, please correct the commands for enable and disable

Anonymitaet · 2019-11-29T11:28:02Z

+
+`isAllowAutoUpdateSchema` can be set via **Pulsar admin API** or **REST API.** 
+
+For how to set `isAllowAutoUpdateSchema` via Pulsar admin API, see [Manage AutoUpdate Strategy](schema-manage.md/#manage-autoupdate-strategy). 


Seems a little weird between #5 and #6? Do we need to add some contents like "If a schema is updated automatically:” after #5?

Anonymitaet

PTAL

congbobo184 · 2019-12-02T10:12:32Z

run cpp tests

congbobo184 · 2019-12-02T10:12:41Z

run Integration Tests

Jennifer88huang-zz · 2019-12-03T14:24:33Z

@congbobo184 @Anonymitaet When you fix an issue, please add the issue number and component in the title, for example, "[Issue 5562][docs] xxx", and add "Fixes #xxxx" in the description.
So when the PR is merged, the corresponding issue will be closed automatically.

sijie · 2019-12-03T17:53:01Z

@congbobo184 @Anonymitaet When you fix an issue, please add the issue number and component in the title, for example, "[Issue 5562][docs] xxx", and add "Fixes #xxxx" in the description.
So when the PR is merged, the corresponding issue will be closed automatically.

maybe we can add this to the contributor guide and encourage committers and contributors following the same practice.

congbobo added 7 commits October 28, 2019 15:38

no message

cdf7735

Revert "no message"

145f97e

This reverts commit cdf7735.

Merge remote-tracking branch 'apache/master'

f27d415

Merge remote-tracking branch 'apache/master'

dc85737

Merge remote-tracking branch 'apache/master'

65e6f83

Merge remote-tracking branch 'apache/master'

260b137

Add

a97e048

sijie assigned congbobo184 Nov 27, 2019

sijie added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. component/schemaregistry labels Nov 27, 2019

sijie added this to the 2.5.0 milestone Nov 27, 2019

congbobo added 5 commits November 28, 2019 10:13

Modify the undifine

8206ddd

Modify the undifine

b5a74d0

modify the undifine

ed9d4e4

Modify the undifine

0ad08a0

modify the style

1626326

Anonymitaet requested changes Nov 29, 2019

View reviewed changes

Anonymitaet reviewed Nov 29, 2019

View reviewed changes

fix some comments

2bacf2d

congbobo added 3 commits November 29, 2019 17:31

Fix some comments

bd8cf19

Fix some comments

2918c0d

recover the image

86751e8

Anonymitaet requested changes Nov 29, 2019

View reviewed changes

Anonymitaet reviewed Nov 29, 2019

View reviewed changes

Anonymitaet requested changes Nov 29, 2019

View reviewed changes

congbobo added 2 commits December 2, 2019 10:33

fix some comments

4573502

Fix some comments

d5dce05

Anonymitaet approved these changes Dec 2, 2019

View reviewed changes

Jennifer88huang-zz changed the title ~~[doc]Modify schema compatibility check doc~~ [Issue 5562][docs]Modify schema compatibility check doc Dec 3, 2019

sijie approved these changes Dec 3, 2019

View reviewed changes

sijie merged commit 180e28a into apache:master Dec 3, 2019


		<td>

		The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.

-The schema compatibility check strategy in order to Upgrade to 2.5.0 or above for compatibility with pulsar version below 2.5.0.
+* If you upgrade a lower version to Pulsar 2.5.0, then it uses the `UNDEFINED` strategy, at this time, it actually uses the compatibility strategy of the topic.
+* If you use Pulsar 2.5.0 and a higher version, then it uses the `UNDEFINED` strategy, at this time, it actually uses the `FULL` compatibility strategy.

+## Schema verification
+When a producer or a consumer tries to connect to a topic, a broker performs some checks to verify a schema.
+### Producer
+When a producer tries to connect to a topic (suppose ignore the schema auto creation), a broker does the following checks:
+* Check if the schema carried by the producer exists in the schema registry or not.
+    * If the schema is already registered, then the producer is connected to a broker and produce messages with that schema.
+    * If the schema is not registered, then Pulsar verifies if the schema is allowed to be registered based on the configured compatibility check strategy.
+### Consumer
+When a consumer tries to connect to a topic, a broker checks if a carried schema is compatible with a registered schema based on the configured schema compatibility check strategy.
+<table style="table">
+<tr>
+<th>
+Compatibility check strategy
+</th>
+<th>
+Check logic
+</th>
+</tr>
+<tr>
+<td>
+`ALWAYS_COMPATIBLE`
+</td>
+<td>
+All pass
+</td>
+</tr>
+<tr>
+<td>
+`ALWAYS_INCOMPATIBLE`
+</td>
+<td>
+No pass
+</td>
+</tr>
+<tr>
+<td>
+`BACKWARD`
+</td>
+<td>
+Can read the last schema
+</td>
+</tr>
+<tr>
+<td>
+`BACKWARD_TRANSITIVE`
+</td>
+<td>
+Can read all schemas
+</td>
+</tr>
+<tr>
+<td>
+`FORWARD`
+</td>
+<td>
+Can read the last schema
+</td>
+</tr>
+<tr>
+<td>
+`FORWARD_TRANSITIVE`
+</td>
+<td>
+Can read the last schema
+</td>
+</tr>
+<tr>
+<td>
+`FULL`
+</td>
+<td>
+Can read the last schema
+</td>
+</tr>
+<tr>
+<td>
+`FULL_TRANSITIVE`
+</td>
+<td>
+Can read all schemas
+</td>
+</tr>
+</table>

	* If the schema is registered, producer is connected.
	* If the schema is registered, a producer is connected to a broker.

	* If the schema isn't registered
	* If the schema is not registered:


		* If the schema isn't registered

		* If`isAllowAutoUpdateSchema` = false, producer is rejected.

	* If`isAllowAutoUpdateSchema` = false, producer is rejected.
	* If `isAllowAutoUpdateSchema` sets to false, the producer is rejected to connect to a broker.


		* If`isAllowAutoUpdateSchema` = false, producer is rejected.

		* If `isAllowAutoUpdateSchema` = true


		* If `isAllowAutoUpdateSchema` = true

		* If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.

	* If schema pass the compatibility check then the broker registers a new schema automatically for the topic and producer is connected.
	* If the schema passes the compatibility check, then the broker registers a new schema automatically for the topic and the producer is connected.

	* If schema does not pass the compatibility, the consumer is rejected.
	* If the schema does not pass the compatibility check, then the consumer is rejected to connect to the broker.

-bin/pulsar-admin namespaces set-is-allow-auto-update-schema --disabled tenant/namespace
+* [Enable AutoUpdate](#enable-autoupdate)
+* [Disable AutoUpdate](#disable-autoupdate)
+* [Adjust compatibility](#adjust-compatibility)
+#### Enable AutoUpdate
+To enable `AutoUpdate` on a namespace, you can use the `pulsar-admin` command.
+```bash
+bin/pulsar-admin namespaces set-is-allow-auto-update-schema --enable tenant/namespace

		5. If no, the broker judge whether this namespace can automatically create schema.
		isAllowAutoUpdateSchema can be modified by Admin-Api and REST-API

	3. If the schema is compatible, the consumer will be connected.
	3. If the schema is compatible, the consumer is connected to the broker.

	4. If the schema is incompatible, the consumer will be disconnected.
	4. If the schema is incompatible, the consumer is disconnected to the broker.

	Pulsar has 9 schema compatibility check strategies, which are summarized in the following table.
	Pulsar has 8 schema compatibility check strategies, which are summarized in the following table.

		* [Enable AutoUpdate](#enable-autoupdate)

		* [Disable AutoUpdate](#disable-autoupdate)


		`isAllowAutoUpdateSchema` can be set via Pulsar admin API or REST API.

		For how to set `isAllowAutoUpdateSchema` via Pulsar admin API, see [Manage AutoUpdate Strategy](schema-manage.md/#manage-autoupdate-strategy).

Uh oh!

Conversation

congbobo184 commented Nov 27, 2019 • edited by Jennifer88huang-zz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Uh oh!

congbobo184 commented Nov 27, 2019

Uh oh!

Anonymitaet commented Nov 27, 2019

Uh oh!

congbobo184 commented Nov 28, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

congbobo184 commented Nov 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sijie commented Nov 29, 2019

Uh oh!

congbobo184 commented Nov 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anonymitaet commented Nov 29, 2019

Uh oh!

congbobo184 commented Nov 29, 2019

Uh oh!

sijie commented Nov 29, 2019

Uh oh!

congbobo184 commented Nov 29, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anonymitaet left a comment

Choose a reason for hiding this comment

Uh oh!

congbobo184 commented Dec 2, 2019

congbobo184 commented Nov 27, 2019 •

edited by Jennifer88huang-zz

Loading

congbobo184 commented Nov 29, 2019 •

edited

Loading

congbobo184 commented Nov 29, 2019 •

edited

Loading