Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 62 additions & 16 deletions site2/docs/schema-understand.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,29 +198,31 @@ Currently, Pulsar supports the following complex types:
| Complex Type | Description |
|---|---|
| `keyvalue` | Represents a complex type of a key/value pair. |
| `struct` | Supports **AVRO**, **JSON**, and **Protobuf**. |
| `struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |

#### keyvalue

`Keyvalue` schema helps applications define schemas for both key and value.

For `SchemaInfo` of `keyvalue` schema, Pulsar stores the `SchemaInfo` of key schema and the `SchemaInfo` of value schema together.

Pulsar provides two methods to encode a key/value pair in messages:
Pulsar provides the following methods to encode a key/value pair in messages:

* `INLINE`

* `SEPARATED`

Users can choose the encoding type when constructing the key/value schema.
You can choose the encoding type when constructing the key/value schema.

##### INLINE
<!--DOCUSAURUS_CODE_TABS-->

Key/value pairs will be encoded together in the message payload.
<!--INLINE-->

##### SEPARATED
Key/value pairs are encoded together in the message payload.

Key will be encoded in the message key and the value will be encoded in the message payload.
<!--SEPARATED-->

Key is encoded in the message key and the value is encoded in the message payload.

**Example**

Expand Down Expand Up @@ -287,25 +289,36 @@ This example shows how to construct a key/value schema and then use it to produc
KeyValue<Integer, String> kv = msg.getValue();
```

<!--END_DOCUSAURUS_CODE_TABS-->

#### struct

Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `struct` schema.
This section describes the details of type and usage of the `struct` schema.

This allows Pulsar:
##### Type

* to use same tools to manage schema definitions
`struct` schema supports `AvroBaseStructSchema` and `ProtobufNativeSchema`.

* to use different serialization/deserialization methods to handle data
|Type|Description|
---|---|
`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br><br>This allows Pulsar:<br>- to use the same tools to manage schema definitions<br>- to use different serialization or deserialization methods to handle data|
`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br><br>This allows Pulsar:<br>- to use native protobuf-v3 to serialize or deserialize data<br>- to use `AutoConsume` to deserialize data.

There are two methods to use `struct` schema:
##### Usage

Pulsar provides the following methods to use the `struct` schema:

* `static`

* `generic`

##### static
* `SchemaDefinition`

<!--DOCUSAURUS_CODE_TABS-->

You can predefine the `struct` schema, and it can be a POJO in Java, a `struct` in Go, or classes generated by Avro or Protobuf tools.
<!--static-->

You can predefine the `struct` schema, which can be a POJO in Java, a `struct` in Go, or classes generated by Avro or Protobuf tools.

**Example**

Expand Down Expand Up @@ -334,7 +347,7 @@ Pulsar gets the schema definition from the predefined `struct` using an Avro lib
User user = consumer.receive();
```

##### generic
<!--generic-->

Sometimes applications do not have pre-defined structs, and you can use this method to define schema and access data.

Expand All @@ -360,6 +373,39 @@ You can define the `struct` schema using the `GenericSchemaBuilder`, generate a
.build()).send();
```

<!--SchemaDefinition-->

You can define the `schemaDefinition` to generate a `struct` schema.

**Example**

1. Create the _User_ class to define the messages sent to Pulsar topics.

```java
public class User {
String name;
int age;
}
```

2. Create a producer with a `SchemaDefinition` and send messages.

```java
SchemaDefinition<User> schemaDefinition = SchemaDefinition.builder().withPojo(User.class).build();
Producer<User> producer = client.newProducer(schemaDefinition).create();
producer.newMessage().value(User.builder().userName("pulsar-user").userId(1L).build()).send();
```

3. Create a consumer with a `SchemaDefinition` schema and receive messages

```java
SchemaDefinition<User> schemaDefinition = SchemaDefinition.builder().withPojo(User.class).build();
Consumer<User> consumer = client.newConsumer(schemaDefinition).subscribe();
User user = consumer.receive();
```

<!--END_DOCUSAURUS_CODE_TABS-->

### Auto Schema

If you don't know the schema type of a Pulsar topic in advance, you can use AUTO schema to produce or consume generic records to or from brokers.
Expand Down Expand Up @@ -399,7 +445,7 @@ pulsarProducer.produce(kafkaMessageBytes);

`AUTO_CONSUME` schema helps a Pulsar topic validate whether the bytes sent by a Pulsar topic is compatible with a consumer, that is, the Pulsar topic deserializes messages into language-specific objects using the `SchemaInfo` retrieved from broker-side.

Currently, `AUTO_CONSUME` only supports **AVRO** and **JSON** schemas. It deserializes messages into `GenericRecord`.
Currently, `AUTO_CONSUME` supports AVRO, JSON and ProtobufNativeSchema schemas. It deserializes messages into `GenericRecord`.

**Example**

Expand Down