From b36d00525ac5d23bc25cbce57e64b00fe912fa55 Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Mon, 18 Oct 2021 01:01:45 +0800 Subject: [PATCH 1/8] doc migration version 2.7.2 chapter Pulsar Schema --- .../schema-evolution-compatibility.md | 205 ++++++ .../version-2.7.2/schema-get-started.md | 102 +++ .../version-2.7.2/schema-manage.md | 686 ++++++++++++++++++ .../version-2.7.2/schema-understand.md | 481 ++++++++++++ .../version-2.7.2-sidebars.json | 22 + 5 files changed, 1496 insertions(+) create mode 100644 site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/schema-manage.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/schema-understand.md diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md b/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md new file mode 100644 index 0000000000000..60a436cb4b720 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md @@ -0,0 +1,205 @@ +--- +id: schema-evolution-compatibility +title: Schema evolution and compatibility +sidebar_label: Schema evolution and compatibility +original_id: schema-evolution-compatibility +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +Normally, schemas do not stay the same over a long period of time. Instead, they undergo evolutions to satisfy new needs. + +This chapter examines how Pulsar schema evolves and what Pulsar schema compatibility check strategies are. + +## Schema evolution + +Pulsar schema is defined in a data structure called `SchemaInfo`. + +Each `SchemaInfo` stored with a topic has a version. The version is used to manage the schema changes happening within a topic. + +The message produced with `SchemaInfo` is tagged with a schema version. When a message is consumed by a Pulsar client, the Pulsar client can use the schema version to retrieve the corresponding `SchemaInfo` and use the correct schema information to deserialize data. + +### What is schema evolution? + +Schemas store the details of attributes and types. To satisfy new business requirements, you need to update schemas inevitably over time, which is called **schema evolution**. + +Any schema changes affect downstream consumers. Schema evolution ensures that the downstream consumers can seamlessly handle data encoded with both old schemas and new schemas. + +### How Pulsar schema should evolve? + +The answer is Pulsar schema compatibility check strategy. It determines how schema compares old schemas with new schemas in topics. + +For more information, see [Schema compatibility check strategy](#schema-compatibility-check-strategy). + +### How does Pulsar support schema evolution? + +1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check. + + The schema compatibility checker is one instance per schema type. + + Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution. + +2. The producer/consumer/reader sends its client `SchemaInfo` to the broker. + +3. The broker knows the schema type and locates the schema compatibility checker for that type. + +4. The broker uses the checker to check if the `SchemaInfo` is compatible with the latest schema of the topic by applying its compatibility check strategy. + + Currently, the compatibility check strategy is configured at the namespace level and applied to all the topics within that namespace. + +## Schema compatibility check strategy + +Pulsar has 8 schema compatibility check strategies, which are summarized in the following table. + +Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is the oldest and V3 is the latest: + +| Compatibility check strategy | Definition | Changes allowed | Check against which schema | Upgrade first | +| --- | --- | --- | --- | --- | +| `ALWAYS_COMPATIBLE` | Disable schema compatibility check. | All changes are allowed | All previous versions | Any order | +| `ALWAYS_INCOMPATIBLE` | Disable schema evolution. | All changes are disabled | None | None | +| `BACKWARD` | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. | * Add optional fields * Delete fields | Latest version | Consumers | +| `BACKWARD_TRANSITIVE` | Consumers using the schema V3 can process data written by producers using the schema V3, V2 or V1. | * Add optional fields * Delete fields | All previous versions | Consumers | +| `FORWARD` | Consumers using the schema V3 or V2 can process data written by producers using the schema V3. | * Add fields * Delete optional fields | Latest version | Producers | +| `FORWARD_TRANSITIVE` | Consumers using the schema V3, V2 or V1 can process data written by producers using the schema V3. | * Add fields * Delete optional fields | All previous versions | Producers | +| `FULL` | Backward and forward compatible between the schema V3 and V2. | * Modify optional fields | Latest version | Any order | +| `FULL_TRANSITIVE` | Backward and forward compatible among the schema V3, V2, and V1. | * Modify optional fields | All previous versions | Any order | + +### ALWAYS_COMPATIBLE and ALWAYS_INCOMPATIBLE + +| Compatibility check strategy | Definition | Note | +| --- | --- | --- | +| `ALWAYS_COMPATIBLE` | Disable schema compatibility check. | None | +| `ALWAYS_INCOMPATIBLE` | Disable schema evolution, that is, any schema change is rejected. | * For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`. * For Avro and JSON, the default schema compatibility check strategy is `FULL`. | + +#### Example + +* Example 1 + + In some situations, an application needs to store events of several different types in the same Pulsar topic. + + In particular, when developing a data model in an `Event Sourcing` style, you might have several kinds of events that affect the state of an entity. + + For example, for a user entity, there are `userCreated`, `userAddressChanged` and `userEnquiryReceived` events. The application requires that those events are always read in the same order. + + Consequently, those events need to go in the same Pulsar partition to maintain order. This application can use `ALWAYS_COMPATIBLE` to allow different kinds of events co-exist in the same topic. + +* Example 2 + + Sometimes we also make incompatible changes. + + For example, you are modifying a field type from `string` to `int`. + + In this case, you need to: + + * Upgrade all producers and consumers to the new schema versions at the same time. + + * Optionally, create a new topic and start migrating applications to use the new topic and the new schema, avoiding the need to handle two incompatible versions in the same topic. + +### BACKWARD and BACKWARD_TRANSITIVE + +Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is the oldest and V3 is the latest: + +| Compatibility check strategy | Definition | Description | +|---|---|---| +`BACKWARD` | Consumers using the new schema can process data written by producers using the **last schema**. | The consumers using the schema V3 can process data written by producers using the schema V3 or V2. | +`BACKWARD_TRANSITIVE` | Consumers using the new schema can process data written by producers using **all previous schemas**. | The consumers using the schema V3 can process data written by producers using the schema V3, V2, or V1. | + +#### Example + +* Example 1 + + Remove a field. + + A consumer constructed to process events without one field can process events written with the old schema containing the field, and the consumer will ignore that field. + +* Example 2 + + You want to load all Pulsar data into a Hive data warehouse and run SQL queries against the data. + + Same SQL queries must continue to work even the data is changed. To support it, you can evolve the schemas using the `BACKWARD` strategy. + +### FORWARD and FORWARD_TRANSITIVE + +Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is the oldest and V3 is the latest: + +| Compatibility check strategy | Definition | Description | +|---|---|---| +`FORWARD` | Consumers using the **last schema** can process data written by producers using a new schema, even though they may not be able to use the full capabilities of the new schema. | The consumers using the schema V3 or V2 can process data written by producers using the schema V3. | +`FORWARD_TRANSITIVE` | Consumers using **all previous schemas** can process data written by producers using a new schema. | The consumers using the schema V3, V2, or V1 can process data written by producers using the schema V3. + +#### Example + +* Example 1 + + Add a field. + + In most data formats, consumers written to process events without new fields can continue doing so even when they receive new events containing new fields. + +* Example 2 + + If a consumer has an application logic tied to a full version of a schema, the application logic may not be updated instantly when the schema evolves. + + In this case, you need to project data with a new schema onto an old schema that the application understands. + + Consequently, you can evolve the schemas using the `FORWARD` strategy to ensure that the old schema can process data encoded with the new schema. + +### FULL and FULL_TRANSITIVE + +Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is the oldest and V3 is the latest: + +| Compatibility check strategy | Definition | Description | Note | +| --- | --- | --- | --- | +| `FULL` | Schemas are both backward and forward compatible, which means: Consumers using the last schema can process data written by producers using the new schema. AND Consumers using the new schema can process data written by producers using the last schema. | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. AND Consumers using the schema V3 or V2 can process data written by producers using the schema V3. | * For Avro and JSON, the default schema compatibility check strategy is `FULL`. * For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`. | +| `FULL_TRANSITIVE` | The new schema is backward and forward compatible with all previously registered schemas. | Consumers using the schema V3 can process data written by producers using the schema V3, V2 or V1. AND Consumers using the schema V3, V2 or V1 can process data written by producers using the schema V3. | None | + +#### Example + +In some data formats, for example, Avro, you can define fields with default values. Consequently, adding or removing a field with a default value is a fully compatible change. + +## Schema verification + +When a producer or a consumer tries to connect to a topic, a broker performs some checks to verify a schema. + +### Producer + +When a producer tries to connect to a topic (suppose ignore the schema auto creation), a broker does the following checks: + +* Check if the schema carried by the producer exists in the schema registry or not. + + * If the schema is already registered, then the producer is connected to a broker and produce messages with that schema. + + * If the schema is not registered, then Pulsar verifies if the schema is allowed to be registered based on the configured compatibility check strategy. + +### Consumer +When a consumer tries to connect to a topic, a broker checks if a carried schema is compatible with a registered schema based on the configured schema compatibility check strategy. + +| Compatibility check strategy | Check logic | +| --- | --- | +| `ALWAYS_COMPATIBLE` | All pass | +| `ALWAYS_INCOMPATIBLE` | No pass | +| `BACKWARD` | Can read the last schema | +| `BACKWARD_TRANSITIVE` | Can read all schemas | +| `FORWARD` | Can read the last schema | +| `FORWARD_TRANSITIVE` | Can read the last schema | +| `FULL` | Can read the last schema | +| `FULL_TRANSITIVE` | Can read all schemas | + +## Order of upgrading clients + +The order of upgrading client applications is determined by the compatibility check strategy. + +For example, the producers using schemas to write data to Pulsar and the consumers using schemas to read data from Pulsar. + +| Compatibility check strategy | Upgrade first | Description | +| --- | --- | --- | +| `ALWAYS_COMPATIBLE` | Any order | The compatibility check is disabled. Consequently, you can upgrade the producers and consumers in **any order**. | +| `ALWAYS_INCOMPATIBLE` | None | The schema evolution is disabled. | +| * `BACKWARD` * `BACKWARD_TRANSITIVE` | Consumers | There is no guarantee that consumers using the old schema can read data produced using the new schema. Consequently, **upgrade all consumers first**, and then start producing new data. | +| * `FORWARD` * `FORWARD_TRANSITIVE` | Producers | There is no guarantee that consumers using the new schema can read data produced using the old schema. Consequently, **upgrade all producers first** to use the new schema and ensure that the data already produced using the old schemas are not available to consumers, and then upgrade the consumers. | +| * `FULL` * `FULL_TRANSITIVE` | Any order | There is no guarantee that consumers using the old schema can read data produced using the new schema and consumers using the new schema can read data produced using the old schema. Consequently, you can upgrade the producers and consumers in **any order**. | + + + + diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md new file mode 100644 index 0000000000000..496fe9077e491 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md @@ -0,0 +1,102 @@ +--- +id: schema-get-started +title: Get started +sidebar_label: Get started +original_id: schema-get-started +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This chapter introduces Pulsar schemas and explains why they are important. + +## Schema Registry + +Type safety is extremely important in any application built around a message bus like Pulsar. + +Producers and consumers need some kind of mechanism for coordinating types at the topic level to avoid various potential problems arise. For example, serialization and deserialization issues. + +Applications typically adopt one of the following approaches to guarantee type safety in messaging. Both approaches are available in Pulsar, and you're free to adopt one or the other or to mix and match on a per-topic basis. + +#### Note +> +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). + +### Client-side approach + +Producers and consumers are responsible for not only serializing and deserializing messages (which consist of raw bytes) but also "knowing" which types are being transmitted via which topics. + +If a producer is sending temperature sensor data on the topic `topic-1`, consumers of that topic will run into trouble if they attempt to parse that data as moisture sensor readings. + +Producers and consumers can send and receive messages consisting of raw byte arrays and leave all type safety enforcement to the application on an "out-of-band" basis. + +### Server-side approach + +Producers and consumers inform the system which data types can be transmitted via the topic. + +With this approach, the messaging system enforces type safety and ensures that producers and consumers remain synced. + +Pulsar has a built-in **schema registry** that enables clients to upload data schemas on a per-topic basis. Those schemas dictate which data types are recognized as valid for that topic. + +## Why use schema + +When a schema is enabled, Pulsar does parse data, it takes bytes as inputs and sends bytes as outputs. While data has meaning beyond bytes, you need to parse data and might encounter parse exceptions which mainly occur in the following situations: + +* The field does not exist + +* The field type has changed (for example, `string` is changed to `int`) + +There are a few methods to prevent and overcome these exceptions, for example, you can catch exceptions when parsing errors, which makes code hard to maintain; or you can adopt a schema management system to perform schema evolution, not to break downstream applications, and enforces type safety to max extend in the language you are using, the solution is Pulsar Schema. + +Pulsar schema enables you to use language-specific types of data when constructing and handling messages from simple types like `string` to more complex application-specific types. + +**Example** + +You can use the _User_ class to define the messages sent to Pulsar topics. + +``` +public class User { + String name; + int age; +} + +``` + +When constructing a producer with the _User_ class, you can specify a schema or not as below. + +### Without schema + +If you construct a producer without specifying a schema, then the producer can only produce messages of type `byte[]`. If you have a POJO class, you need to serialize the POJO into bytes before sending messages. + +**Example** + +``` +Producer producer = client.newProducer() + .topic(topic) + .create(); +User user = new User("Tom", 28); +byte[] message = … // serialize the `user` by yourself; +producer.send(message); + +``` +### With schema + +If you construct a producer with specifying a schema, then you can send a class to a topic directly without worrying about how to serialize POJOs into bytes. + +**Example** + +This example constructs a producer with the _JSONSchema_, and you can send the _User_ class to topics directly without worrying about how to serialize it into bytes. + +``` +Producer producer = client.newProducer(JSONSchema.of(User.class)) + .topic(topic) + .create(); +User user = new User("Tom", 28); +producer.send(user); + +``` + +### Summary + +When constructing a producer with a schema, you do not need to serialize messages into bytes, instead Pulsar schema does this job in the background. diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md b/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md new file mode 100644 index 0000000000000..aa42485736939 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md @@ -0,0 +1,686 @@ +--- +id: schema-manage +title: Manage schema +sidebar_label: Manage schema +original_id: schema-manage +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This guide demonstrates the ways to manage schemas: + +* Automatically + + * [Schema AutoUpdate](#schema-autoupdate) + +* Manually + + * [Schema manual management](#schema-manual-management) + + * [Custom schema storage](#custom-schema-storage) + +## Schema AutoUpdate + +If a schema passes the schema compatibility check, Pulsar producer automatically updates this schema to the topic it produces by default. + +### AutoUpdate for producer + +For a producer, the `AutoUpdate` happens in the following cases: + +* If a **topic doesn’t have a schema**, Pulsar registers a schema automatically. + +* If a **topic has a schema**: + + * If a **producer doesn’t carry a schema**: + + * If `isSchemaValidationEnforced` or `schemaValidationEnforced` is **disabled** in the namespace to which the topic belongs, the producer is allowed to connect to the topic and produce data. + + * If `isSchemaValidationEnforced` or `schemaValidationEnforced` is **enabled** in the namespace to which the topic belongs, the producer is rejected and disconnected. + + * If a **producer carries a schema**: + + A broker performs the compatibility check based on the configured compatibility check strategy of the namespace to which the topic belongs. + + * If the schema is registered, a producer is connected to a broker. + + * If the schema is not registered: + + * If `isAllowAutoUpdateSchema` sets to **false**, the producer is rejected to connect to a broker. + + * If `isAllowAutoUpdateSchema` sets to **true**: + + * If the schema passes the compatibility check, then the broker registers a new schema automatically for the topic and the producer is connected. + + * If the schema does not pass the compatibility check, then the broker does not register a schema and the producer is rejected to connect to a broker. + +![AutoUpdate Producer](/assets/schema-producer.png) + +### AutoUpdate for consumer + +For a consumer, the `AutoUpdate` happens in the following cases: + +* If a **consumer connects to a topic without a schema** (which means the consumer receiving raw bytes), the consumer can connect to the topic successfully without doing any compatibility check. + +* If a **consumer connects to a topic with a schema**. + + * If a topic does not have all of them (a schema/data/a local consumer and a local producer): + + * If `isAllowAutoUpdateSchema` sets to **true**, then the consumer registers a schema and it is connected to a broker. + + * If `isAllowAutoUpdateSchema` sets to **false**, then the consumer is rejected to connect to a broker. + + * If a topic has one of them (a schema/data/a local consumer and a local producer), then the schema compatibility check is performed. + + * If the schema passes the compatibility check, then the consumer is connected to the broker. + + * If the schema does not pass the compatibility check, then the consumer is rejected to connect to the broker. + +![AutoUpdate Consumer](/assets/schema-consumer.png) + + +### Manage AutoUpdate strategy + +You can use the `pulsar-admin` command to manage the `AutoUpdate` strategy as below: + +* [Enable AutoUpdate](#enable-autoupdate) + +* [Disable AutoUpdate](#disable-autoupdate) + +* [Adjust compatibility](#adjust-compatibility) + +#### Enable AutoUpdate + +To enable `AutoUpdate` on a namespace, you can use the `pulsar-admin` command. + +```bash + +bin/pulsar-admin namespaces set-is-allow-auto-update-schema --enable tenant/namespace + +``` + +#### Disable AutoUpdate + +To disable `AutoUpdate` on a namespace, you can use the `pulsar-admin` command. + +```bash + +bin/pulsar-admin namespaces set-is-allow-auto-update-schema --disable tenant/namespace + +``` + +Once the `AutoUpdate` is disabled, you can only register a new schema using the `pulsar-admin` command. + +#### Adjust compatibility + +To adjust the schema compatibility level on a namespace, you can use the `pulsar-admin` command. + +```bash + +bin/pulsar-admin namespaces set-schema-compatibility-strategy --compatibility tenant/namespace + +``` + +### Schema validation + +By default, `schemaValidationEnforced` is **disabled** for producers: + +* This means a producer without a schema can produce any kind of messages to a topic with schemas, which may result in producing trash data to the topic. + +* This allows non-java language clients that don’t support schema can produce messages to a topic with schemas. + +However, if you want a stronger guarantee on the topics with schemas, you can enable `schemaValidationEnforced` across the whole cluster or on a per-namespace basis. + +#### Enable schema validation + +To enable `schemaValidationEnforced` on a namespace, you can use the `pulsar-admin` command. + +```bash + +bin/pulsar-admin namespaces set-schema-validation-enforce --enable tenant/namespace + +``` + +#### Disable schema validation + +To disable `schemaValidationEnforced` on a namespace, you can use the `pulsar-admin` command. + +```bash + +bin/pulsar-admin namespaces set-schema-validation-enforce --disable tenant/namespace + +``` + +## Schema manual management + +To manage schemas, you can use one of the following methods. + +| Method | Description | +| --- | --- | +| **Admin CLI** | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **REST API** | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | +| **Java Admin API** | Pulsar provides Java admin library. | + +### Upload a schema + +To upload (register) a new schema for a topic, you can use one of the following methods. + + + + + +Use the `upload` subcommand. + +```bash + +$ pulsar-admin schemas upload --filename + +``` + +The `schema-definition-file` is in JSON format. + +```json + +{ + "type": "", + "schema": "", + "properties": {} // the properties associated with the schema +} + +``` + +The `schema-definition-file` includes the following fields: + +| Field | Description | +| --- | --- | +| `type` | The schema type. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `properties` | The additional properties associated with the schema. | + +Here are examples of the `schema-definition-file` for a JSON schema. + +**Example 1** + +```json + +{ + "type": "JSON", + "schema": "{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"com.foo\",\"fields\":[{\"name\":\"file1\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"file2\",\"type\":\"string\",\"default\":null},{\"name\":\"file3\",\"type\":[\"null\",\"string\"],\"default\":\"dfdf\"}]}", + "properties": {} +} + +``` + +**Example 2** + +```json + +{ + "type": "STRING", + "schema": "", + "properties": { + "key1": "value1" + } +} + +``` + + + + +Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v2/schemas/:tenant/:namespace/:topic/schema|operation/uploadSchem?version=@pulsar:version_number@a} + +The post payload is in JSON format. + +```json + +{ + "type": "", + "schema": "", + "properties": {} // the properties associated with the schema +} + +``` + +The post payload includes the following fields: + +| Field | Description | +| --- | --- | +| `type` | The schema type. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `properties` | The additional properties associated with the schema. | + + + + +```java + +void createSchema(String topic, PostSchemaPayload schemaPayload) + +``` + +The `PostSchemaPayload` includes the following fields: + +| Field | Description | +| --- | --- | +| `type` | The schema type. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `properties` | The additional properties associated with the schema. | + +Here is an example of `PostSchemaPayload`: + +```java + +PulsarAdmin admin = …; + +PostSchemaPayload payload = new PostSchemaPayload(); +payload.setType("INT8"); +payload.setSchema(""); + +admin.createSchema("my-tenant/my-ns/my-topic", payload); + +``` + + + + +### Get a schema (latest) + +To get the latest schema for a topic, you can use one of the following methods. + + + + + +Use the `get` subcommand. + +```bash + +$ pulsar-admin schemas get + +{ + "version": 0, + "type": "String", + "timestamp": 0, + "data": "string", + "properties": { + "property1": "string", + "property2": "string" + } +} + +``` + + + + +Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v2/schemas/:tenant/:namespace/:topic/schema|operation/getSchem?version=@pulsar:version_number@a} + +Here is an example of a response, which is returned in JSON format. + +```json + +{ + "version": "", + "type": "", + "timestamp": "", + "data": "", + "properties": {} // the properties associated with the schema +} + +``` + +The response includes the following fields: + +| Field | Description | +| --- | --- | +| `version` | The schema version, which is a long number. | +| `type` | The schema type. | +| `timestamp` | The timestamp of creating this version of schema. | +| `data` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `properties` | The additional properties associated with the schema. | + + + + +```java + +SchemaInfo createSchema(String topic) + +``` + +The `SchemaInfo` includes the following fields: + +| Field | Description | +| --- | --- | +| `name` | The schema name. | +| `type` | The schema type. | +| `schema` | A byte array of the schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this byte array should be empty. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition converted to a byte array. | +| `properties` | The additional properties associated with the schema. | + +Here is an example of `SchemaInfo`: + +```java + +PulsarAdmin admin = …; + +SchemaInfo si = admin.getSchema("my-tenant/my-ns/my-topic"); + +``` + + + + + +### Get a schema (specific) + +To get a specific version of a schema, you can use one of the following methods. + + + + + +Use the `get` subcommand. + +```bash + +$ pulsar-admin schemas get --version= + +``` + + + + +Send a `GET` request to a schema endpoint: {@inject: endpoint|GET|/admin/v2/schemas/:tenant/:namespace/:topic/schema/:version|operation/getSchem?version=@pulsar:version_number@a} + +Here is an example of a response, which is returned in JSON format. + +```json + +{ + "version": "", + "type": "", + "timestamp": "", + "data": "", + "properties": {} // the properties associated with the schema +} + +``` + +The response includes the following fields: + +| Field | Description | +| --- | --- | +| `version` | The schema version, which is a long number. | +| `type` | The schema type. | +| `timestamp` | The timestamp of creating this version of schema. | +| `data` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `properties` | The additional properties associated with the schema. | + + + + +```java + +SchemaInfo createSchema(String topic, long version) + +``` + +The `SchemaInfo` includes the following fields: + +| Field | Description | +| --- | --- | +| `name` | The schema name. | +| `type` | The schema type. | +| `schema` | A byte array of the schema definition data, which is encoded in UTF 8. * If the schema is a **primitive** schema, this byte array should be empty. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition converted to a byte array. | +| `properties` | The additional properties associated with the schema. | + +Here is an example of `SchemaInfo`: + +```java + +PulsarAdmin admin = …; + +SchemaInfo si = admin.getSchema("my-tenant/my-ns/my-topic", 1L); + +``` + + + + + +### Extract a schema + +To provide a schema via a topic, you can use the following method. + + + + + +Use the `extract` subcommand. + +```bash + +$ pulsar-admin schemas extract --classname --jar --type + +``` + + + + + +### Delete a schema + +To delete a schema for a topic, you can use one of the following methods. + +:::note + + +In any case, the **delete** action deletes **all versions** of a schema registered for a topic. + +::: + + + + + +Use the `delete` subcommand. + +```bash + +$ pulsar-admin schemas delete + +``` + + + + +Send a `DELETE` request to a schema endpoint: {@inject: endpoint|DELETE|/admin/v2/schemas/:tenant/:namespace/:topic/schema|operation/deleteSchema?version=@pulsar:version_number@} + +Here is an example of a response, which is returned in JSON format. + +```json + +{ + "version": "", +} + +``` + +The response includes the following field: + +Field | Description | +---|---| +`version` | The schema version, which is a long number. | + + + + +```java + +void deleteSchema(String topic) + +``` + +Here is an example of deleting a schema. + +```java + +PulsarAdmin admin = …; + +admin.deleteSchema("my-tenant/my-ns/my-topic"); + +``` + + + + + +## Custom schema storage + +By default, Pulsar stores various data types of schemas in [Apache BookKeeper](https://bookkeeper.apache.org) deployed alongside Pulsar. + +However, you can use another storage system if needed. + +### Implement + +To use a non-default (non-BookKeeper) storage system for Pulsar schemas, you need to implement the following Java interfaces: + +* [SchemaStorage interface](#schemastorage-interface) + +* [SchemaStorageFactory interface](#schemastoragefactory-interface) + +#### SchemaStorage interface + +The `SchemaStorage` interface has the following methods: + +```java + +public interface SchemaStorage { + // How schemas are updated + CompletableFuture put(String key, byte[] value, byte[] hash); + + // How schemas are fetched from storage + CompletableFuture get(String key, SchemaVersion version); + + // How schemas are deleted + CompletableFuture delete(String key); + + // Utility method for converting a schema version byte array to a SchemaVersion object + SchemaVersion versionFromBytes(byte[] version); + + // Startup behavior for the schema storage client + void start() throws Exception; + + // Shutdown behavior for the schema storage client + void close() throws Exception; +} + +``` + +:::tip + + +For a complete example of **schema storage** implementation, see [BookKeeperSchemaStorage](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java) class. + +::: + +#### SchemaStorageFactory interface + +The `SchemaStorageFactory` interface has the following method: + +```java + +public interface SchemaStorageFactory { + @NotNull + SchemaStorage create(PulsarService pulsar) throws Exception; +} + +``` + +:::tip + + +For a complete example of **schema storage factory** implementation, see [BookKeeperSchemaStorageFactory](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorageFactory.java) class. + +::: + +### Deploy + +To use your custom schema storage implementation, perform the following steps. + +1. Package the implementation in a [JAR](https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html) file. + +2. Add the JAR file to the `lib` folder in your Pulsar binary or source distribution. + +3. Change the `schemaRegistryStorageClassName` configuration in `broker.conf` to your custom factory class. + +4. Start Pulsar. diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md b/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md new file mode 100644 index 0000000000000..7de41e6c3838f --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md @@ -0,0 +1,481 @@ +--- +id: schema-understand +title: Understand schema +sidebar_label: Understand schema +original_id: schema-understand +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This chapter explains the basic concepts of Pulsar schema, focuses on the topics of particular importance, and provides additional background. + +## SchemaInfo + +Pulsar schema is defined in a data structure called `SchemaInfo`. + +The `SchemaInfo` is stored and enforced on a per-topic basis and cannot be stored at the namespace or tenant level. + +A `SchemaInfo` consists of the following fields: + +| Field | Description | +| --- | --- | +| `name` | Schema name (a string). | +| `type` | Schema type, which determines how to interpret the schema data. * Predefined schema: see [here](schema-understand.md#schema-type). * Customized schema: it is left as an empty string. | +| `schema`（`payload`) | Schema data, which is a sequence of 8-bit unsigned bytes and schema-type specific. | +| `properties` | It is a user defined properties as a string/string map. Applications can use this bag for carrying any application specific logics. Possible properties might be the Git hash associated with the schema, an environment string like `dev` or `prod`. | + +**Example** + +This is the `SchemaInfo` of a string. + +```json + +{ + "name": "test-string-schema", + "type": "STRING", + "schema": "", + "properties": {} +} + +``` + +## Schema type + +Pulsar supports various schema types, which are mainly divided into two categories: + +* Primitive type + +* Complex type + +### Primitive type + +Currently, Pulsar supports the following primitive types: + +| Primitive Type | Description | +|---|---| +| `BOOLEAN` | A binary value | +| `INT8` | A 8-bit signed integer | +| `INT16` | A 16-bit signed integer | +| `INT32` | A 32-bit signed integer | +| `INT64` | A 64-bit signed integer | +| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | +| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | +| `BYTES` | A sequence of 8-bit unsigned bytes | +| `STRING` | A Unicode character sequence | +| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision.
It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | +| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| +| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| +| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| +| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | + +For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` is used to determine how to serialize and deserialize the data. + +Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings. + +The conversions between **Pulsar schema types** and **language-specific primitive types** are as below. + +| Schema Type | Java Type| Python Type | Go Type | +|---|---|---|---| +| BOOLEAN | boolean | bool | bool | +| INT8 | byte | | int8 | +| INT16 | short | | int16 | +| INT32 | int | | int32 | +| INT64 | long | | int64 | +| FLOAT | float | float | float32 | +| DOUBLE | double | float | float64| +| BYTES | byte[], ByteBuffer, ByteBuf | bytes | []byte | +| STRING | string | str | string| +| TIMESTAMP | java.sql.Timestamp | | | +| TIME | java.sql.Time | | | +| DATE | java.util.Date | | | +| INSTANT | java.time.Instant | | | +| LOCAL_DATE | java.time.LocalDate | | | +| LOCAL_TIME | java.time.LocalTime | | +| LOCAL_DATE_TIME | java.time.LocalDateTime | | + +**Example** + +This example demonstrates how to use a string schema. + +1. Create a producer with a string schema and send messages. + + ```java + + Producer producer = client.newProducer(Schema.STRING).create(); + producer.newMessage().value("Hello Pulsar!").send(); + + ``` + +2. Create a consumer with a string schema and receive messages. + + ```java + + Consumer consumer = client.newConsumer(Schema.STRING).subscribe(); + consumer.receive(); + + ``` + +### Complex type + +Currently, Pulsar supports the following complex types: + +| Complex Type | Description | +|---|---| +| `keyvalue` | Represents a complex type of a key/value pair. | +| `struct` | Supports **AVRO**, **JSON**, and **Protobuf**. | + +#### keyvalue + +`Keyvalue` schema helps applications define schemas for both key and value. + +For `SchemaInfo` of `keyvalue` schema, Pulsar stores the `SchemaInfo` of key schema and the `SchemaInfo` of value schema together. + +Pulsar provides two methods to encode a key/value pair in messages： + +* `INLINE` + +* `SEPARATED` + +Users can choose the encoding type when constructing the key/value schema. + +##### INLINE + +Key/value pairs will be encoded together in the message payload. + +##### SEPARATED + +Key will be encoded in the message key and the value will be encoded in the message payload. + +**Example** + +This example shows how to construct a key/value schema and then use it to produce and consume messages. + +1. Construct a key/value schema with `INLINE` encoding type. + + ```java + + Schema> kvSchema = Schema.KeyValue( + Schema.INT32, + Schema.STRING, + KeyValueEncodingType.INLINE + ); + + ``` + +2. Optionally, construct a key/value schema with `SEPARATED` encoding type. + + ```java + + Schema> kvSchema = Schema.KeyValue( + Schema.INT32, + Schema.STRING, + KeyValueEncodingType.SEPARATED + ); + + ``` + +3. Produce messages using a key/value schema. + + ```java + + Schema> kvSchema = Schema.KeyValue( + Schema.INT32, + Schema.STRING, + KeyValueEncodingType.SEPARATED + ); + + Producer> producer = client.newProducer(kvSchema) + .topic(TOPIC) + .create(); + + final int key = 100; + final String value = "value-100"; + + // send the key/value message + producer.newMessage() + .value(new KeyValue(key, value)) + .send(); + + ``` + +4. Consume messages using a key/value schema. + + ```java + + Schema> kvSchema = Schema.KeyValue( + Schema.INT32, + Schema.STRING, + KeyValueEncodingType.SEPARATED + ); + + Consumer> consumer = client.newConsumer(kvSchema) + ... + .topic(TOPIC) + .subscriptionName(SubscriptionName).subscribe(); + + // receive key/value pair + Message> msg = consumer.receive(); + KeyValue kv = msg.getValue(); + + ``` + +#### struct + +Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `struct` schema. + +This allows Pulsar: + +* to use same tools to manage schema definitions + +* to use different serialization/deserialization methods to handle data + +There are two methods to use `struct` schema： + +* `static` + +* `generic` + +##### static + +You can predefine the `struct` schema, and it can be a POJO in Java, a `struct` in Go, or classes generated by Avro or Protobuf tools. + +**Example** + +Pulsar gets the schema definition from the predefined `struct` using an Avro library. The schema definition is the schema data stored as a part of the `SchemaInfo`. + +1. Create the _User_ class to define the messages sent to Pulsar topics. + + ```java + + public class User { + String name; + int age; + } + + ``` + +2. Create a producer with a `struct` schema and send messages. + + ```java + + Producer producer = client.newProducer(Schema.AVRO(User.class)).create(); + producer.newMessage().value(User.builder().userName("pulsar-user").userId(1L).build()).send(); + + ``` + +3. Create a consumer with a `struct` schema and receive messages + + ```java + + Consumer consumer = client.newConsumer(Schema.AVRO(User.class)).subscribe(); + User user = consumer.receive(); + + ``` + +##### generic + +Sometimes applications do not have pre-defined structs, and you can use this method to define schema and access data. + +You can define the `struct` schema using the `GenericSchemaBuilder`, generate a generic struct using `GenericRecordBuilder` and consume messages into `GenericRecord`. + +**Example** + +1. Use `RecordSchemaBuilder` to build a schema. + + ```java + + RecordSchemaBuilder recordSchemaBuilder = SchemaBuilder.record("schemaName"); + recordSchemaBuilder.field("intField").type(SchemaType.INT32); + SchemaInfo schemaInfo = recordSchemaBuilder.build(SchemaType.AVRO); + + Producer producer = client.newProducer(Schema.generic(schemaInfo)).create(); + + ``` + +2. Use `RecordBuilder` to build the struct records. + + ```java + + producer.newMessage().value(schema.newRecordBuilder() + .set("intField", 32) + .build()).send(); + + ``` + +### Auto Schema + +If you don't know the schema type of a Pulsar topic in advance, you can use AUTO schema to produce or consume generic records to or from brokers. + +| Auto Schema Type | Description | +|---|---| +| `AUTO_PRODUCE` | This is useful for transferring data **from a producer to a Pulsar topic that has a schema**. | +| `AUTO_CONSUME` | This is useful for transferring data **from a Pulsar topic that has a schema to a consumer**. | + +#### AUTO_PRODUCE + +`AUTO_PRODUCE` schema helps a producer validate whether the bytes sent by the producer is compatible with the schema of a topic. + +**Example** + +Suppose that: + +* You have a producer processing messages from a Kafka topic _K_. + +* You have a Pulsar topic _P_, and you do not know its schema type. + +* Your application reads the messages from _K_ and writes the messages to _P_. + +In this case, you can use `AUTO_PRODUCE` to verify whether the bytes produced by _K_ can be sent to _P_ or not. + +```java + +Produce pulsarProducer = client.newProducer(Schema.AUTO_PRODUCE()) + … + .create(); + +byte[] kafkaMessageBytes = … ; + +pulsarProducer.produce(kafkaMessageBytes); + +``` + +#### AUTO_CONSUME + +`AUTO_CONSUME` schema helps a Pulsar topic validate whether the bytes sent by a Pulsar topic is compatible with a consumer, that is, the Pulsar topic deserializes messages into language-specific objects using the `SchemaInfo` retrieved from broker-side. + +Currently, `AUTO_CONSUME` only supports **AVRO** and **JSON** schemas. It deserializes messages into `GenericRecord`. + +**Example** + +Suppose that: + +* You have a Pulsar topic _P_. + +* You have a consumer (for example, MySQL) receiving messages from the topic _P_. + +* You application reads the messages from _P_ and writes the messages to MySQL. + +In this case, you can use `AUTO_CONSUME` to verify whether the bytes produced by _P_ can be sent to MySQL or not. + +```java + +Consumer pulsarConsumer = client.newConsumer(Schema.AUTO_CONSUME()) + … + .subscribe(); + +Message msg = consumer.receive() ; +GenericRecord record = msg.getValue(); + +``` + +## Schema version + +Each `SchemaInfo` stored with a topic has a version. Schema version manages schema changes happening within a topic. + +Messages produced with a given `SchemaInfo` is tagged with a schema version, so when a message is consumed by a Pulsar client, the Pulsar client can use the schema version to retrieve the corresponding `SchemaInfo` and then use the `SchemaInfo` to deserialize data. + +Schemas are versioned in succession. Schema storage happens in a broker that handles the associated topics so that version assignments can be made. + +Once a version is assigned/fetched to/for a schema, all subsequent messages produced by that producer are tagged with the appropriate version. + +**Example** + +The following example illustrates how the schema version works. + +Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: + +```java + +PulsarClient client = PulsarClient.builder() + .serviceUrl("pulsar://localhost:6650") + .build(); + +Producer producer = client.newProducer(JSONSchema.of(SensorReading.class)) + .topic("sensor-data") + .sendTimeout(3, TimeUnit.SECONDS) + .create(); + +``` + +The table below lists the possible scenarios when this connection attempt occurs and what happens in each scenario: + +| Scenario | What happens | +| --- | --- | +| * No schema exists for the topic. | (1) The producer is created using the given schema. (2) Since no existing schema is compatible with the `SensorReading` schema, the schema is transmitted to the broker and stored. (3) Any consumer created using the same schema or topic can consume messages from the `sensor-data` topic. | +| * A schema already exists. * The producer connects using the same schema that is already stored. | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible. (3) The broker attempts to store the schema in [BookKeeper](concepts-architecture-overview.md#persistent-storage) but then determines that it's already stored, so it is used to tag produced messages. | * A schema already exists. * The producer connects using a new schema that is compatible. | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible and stores the new schema as the current version (with a new version number). | + +## How does schema work + +Pulsar schemas are applied and enforced at the **topic** level (schemas cannot be applied at the namespace or tenant level). + +Producers and consumers upload schemas to brokers, so Pulsar schemas work on the producer side and the consumer side. + +### Producer side + +This diagram illustrates how does schema work on the Producer side. + +![Schema works at the producer side](/assets/schema-producer.png) + +1. The application uses a schema instance to construct a producer instance. + + The schema instance defines the schema for the data being produced using the producer instance. + + Take AVRO as an example, Pulsar extract schema definition from the POJO class and construct the `SchemaInfo` that the producer needs to pass to a broker when it connects. + +2. The producer connects to the broker with the `SchemaInfo` extracted from the passed-in schema instance. + +3. The broker looks up the schema in the schema storage to check if it is already a registered schema. + +4. If yes, the broker skips the schema validation since it is a known schema, and returns the schema version to the producer. + +5. If no, the broker verifies whether a schema can be automatically created in this namespace: + + * If `isAllowAutoUpdateSchema` sets to **true**, then a schema can be created, and the broker validates the schema based on the schema compatibility check strategy defined for the topic. + + * If `isAllowAutoUpdateSchema` sets to **false**, then a schema can not be created, and the producer is rejected to connect to the broker. + +**Tip**: + +`isAllowAutoUpdateSchema` can be set via **Pulsar admin API** or **REST API.** + +For how to set `isAllowAutoUpdateSchema` via Pulsar admin API, see [Manage AutoUpdate Strategy](schema-manage.md/#manage-autoupdate-strategy). + +6. If the schema is allowed to be updated, then the compatible strategy check is performed. + + * If the schema is compatible, the broker stores it and returns the schema version to the producer. + + All the messages produced by this producer are tagged with the schema version. + + * If the schema is incompatible, the broker rejects it. + +### Consumer side + +This diagram illustrates how does Schema work on the consumer side. + +![Schema works at the consumer side](/assets/schema-consumer.png) + +1. The application uses a schema instance to construct a consumer instance. + + The schema instance defines the schema that the consumer uses for decoding messages received from a broker. + +2. The consumer connects to the broker with the `SchemaInfo` extracted from the passed-in schema instance. + +3. The broker determines whether the topic has one of them (a schema/data/a local consumer and a local producer). + +4. If a topic does not have all of them (a schema/data/a local consumer and a local producer): + + * If `isAllowAutoUpdateSchema` sets to **true**, then the consumer registers a schema and it is connected to a broker. + + * If `isAllowAutoUpdateSchema` sets to **false**, then the consumer is rejected to connect to a broker. + +5. If a topic has one of them (a schema/data/a local consumer and a local producer), then the schema compatibility check is performed. + + * If the schema passes the compatibility check, then the consumer is connected to the broker. + + * If the schema does not pass the compatibility check, then the consumer is rejected to connect to the broker. + +6. The consumer receives messages from the broker. + + If the schema used by the consumer supports schema versioning (for example, AVRO schema), the consumer fetches the `SchemaInfo` of the version tagged in messages and uses the passed-in schema and the schema tagged in messages to decode the messages. diff --git a/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json b/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json index 078728029e300..e8c9dbbc07207 100644 --- a/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json +++ b/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json @@ -63,6 +63,28 @@ "id": "version-2.7.2/concepts-multiple-advertised-listeners" } ] + }, + { + "type": "category", + "label": "Pulsar Schema", + "items": [ + { + "type": "doc", + "id": "version-2.7.2/schema-get-started" + }, + { + "type": "doc", + "id": "version-2.7.2/schema-understand" + }, + { + "type": "doc", + "id": "version-2.7.2/schema-evolution-compatibility" + }, + { + "type": "doc", + "id": "version-2.7.2/schema-manage" + } + ] } ] } \ No newline at end of file From 63e50a859616e2741c5b15ee84222ed86e73f3de Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Mon, 18 Oct 2021 01:45:26 +0800 Subject: [PATCH 2/8] doc migration 2.7.2 chapter Pulsar IO --- .../versioned_docs/version-2.7.2/io-cdc.md | 30 + .../versioned_docs/version-2.7.2/io-cli.md | 664 ++++++ .../version-2.7.2/io-connectors.md | 236 ++ .../versioned_docs/version-2.7.2/io-debug.md | 398 ++++ .../version-2.7.2/io-develop.md | 270 +++ .../version-2.7.2/io-overview.md | 176 ++ .../version-2.7.2/io-quickstart.md | 983 +++++++++ .../versioned_docs/version-2.7.2/io-use.md | 1957 +++++++++++++++++ .../version-2.7.2-sidebars.json | 38 + 9 files changed, 4752 insertions(+) create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-cdc.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-cli.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-connectors.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-debug.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-develop.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-overview.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-use.md diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md b/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md new file mode 100644 index 0000000000000..f78686012d45e --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md @@ -0,0 +1,30 @@ +--- +id: io-cdc +title: CDC connector +sidebar_label: CDC connector +original_id: io-cdc +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +CDC source connectors capture log changes of databases (such as MySQL, MongoDB, and PostgreSQL) into Pulsar. + +> CDC source connectors are built on top of [Canal](https://github.com/alibaba/canal) and [Debezium](https://debezium.io/) and store all data into Pulsar cluster in a persistent, replicated, and partitioned way. + +Currently, Pulsar has the following CDC connectors. + +Name|Java Class +|---|--- +[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • + +For more information about Canal and Debezium, see the information below. + +Subject | Reference +|---|--- +How to use Canal source connector with MySQL|[Canal guide](https://github.com/alibaba/canal/wiki) +How does Canal work | [Canal tutorial](https://github.com/alibaba/canal/wiki) +How to use Debezium source connector with MySQL | [Debezium guide](https://debezium.io/docs/connectors/mysql/) +How does Debezium work | [Debezium tutorial](https://debezium.io/docs/tutorial/) diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cli.md b/site2/website-next/versioned_docs/version-2.7.2/io-cli.md new file mode 100644 index 0000000000000..c9c19e4c3c492 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cli.md @@ -0,0 +1,664 @@ +--- +id: io-cli +title: Connector Admin CLI +sidebar_label: CLI +original_id: io-cli +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The `pulsar-admin` tool helps you manage Pulsar connectors. + +## `sources` + +An interface for managing Pulsar IO sources (ingress data into Pulsar). + +```bash + +$ pulsar-admin sources subcommands + +``` + +Subcommands are: + +* `create` + +* `update` + +* `delete` + +* `get` + +* `status` + +* `list` + +* `stop` + +* `start` + +* `restart` + +* `localrun` + +* `available-sources` + +* `reload` + + +### `create` + +Submit a Pulsar IO source connector to run in a Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sources create options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the NAR archive for the source.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--batch-source-config` | BatchSource configuration key/values pairs provided as a JSON string, e.g., { "discoveryTriggererClassName" : "org.apache.pulsar.io.batchdiscovery.CronTriggerer", "discoveryTriggererConfig": {"cron": "*/5 * * * *"} } +| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `--cpu` | The CPU (in cores) that needs to be allocated per source instance (applicable only to Docker runtime). +| `--deserialization-classname` | The SerDe classname for the source. +| `--destination-topic-name` | The Pulsar topic to which data is sent. +| `--disk` | The disk (in bytes) that needs to be allocated per source instance (applicable only to Docker runtime). +|`--name` | The source's name. +| `--namespace` | The source's namespace. +| ` --parallelism` | The source's parallelism factor, that is, the number of source instances to run. +| `--processing-guarantees` | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from external system and writes messages to a Pulsar topic. The `--processing-guarantees` is used to ensure the processing guarantees for writing messages to the Pulsar topic.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +| `--ram` | The RAM (in bytes) that needs to be allocated per source instance (applicable only to the process and Docker runtimes). +| `-st`, `--schema-type` | The schema type.
    Either a builtin schema (for example, AVRO and JSON) or custom schema class name to be used to encode messages emitted from source. +| `--source-config` | Source config key/values. +| `--source-config-file` | The path to a YAML config file specifying the source's configuration. +| `-t`, `--source-type` | The source's connector provider. +| `--tenant` | The source's tenant. +|`--producer-config`| The custom producer configuration (as a JSON string). + +### `update` + +Update a already submitted Pulsar IO source connector. + +#### Usage + +```bash + +$ pulsar-admin sources update options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the NAR archive for the source.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--batch-source-config` | BatchSource configuration key/values pairs provided as a JSON string, e.g., { "discoveryTriggererClassName" : "org.apache.pulsar.io.batchdiscovery.CronTriggerer", "discoveryTriggererConfig": {"cron": "*/5 * * * *"} } +| `--classname` | The source's class name if `archive` is file-url-path (file://). +| `--cpu` | The CPU (in cores) that needs to be allocated per source instance (applicable only to Docker runtime). +| `--deserialization-classname` | The SerDe classname for the source. +| `--destination-topic-name` | The Pulsar topic to which data is sent. +| `--disk` | The disk (in bytes) that needs to be allocated per source instance (applicable only to Docker runtime). +|`--name` | The source's name. +| `--namespace` | The source's namespace. +| ` --parallelism` | The source's parallelism factor, that is, the number of source instances to run. +| `--processing-guarantees` | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from external system and writes messages to a Pulsar topic. The `--processing-guarantees` is used to ensure the processing guarantees for writing messages to the Pulsar topic.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +| `--ram` | The RAM (in bytes) that needs to be allocated per source instance (applicable only to the process and Docker runtimes). +| `-st`, `--schema-type` | The schema type.
    Either a builtin schema (for example, AVRO and JSON) or custom schema class name to be used to encode messages emitted from source. +| `--source-config` | Source config key/values. +| `--source-config-file` | The path to a YAML config file specifying the source's configuration. +| `-t`, `--source-type` | The source's connector provider. The `source-type` parameter of the currently built-in connectors is determined by the setting of the `name` parameter specified in the pulsar-io.yaml file. +| `--tenant` | The source's tenant. +| `--update-auth-data` | Whether or not to update the auth data.
    **Default value: false.** + + +### `delete` + +Delete a Pulsar IO source connector. + +#### Usage + +```bash + +$ pulsar-admin sources delete options + +``` + +#### Option + +|Flag|Description| +|---|---| +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + +### `get` + +Get the information about a Pulsar IO source connector. + +#### Usage + +```bash + +$ pulsar-admin sources get options + +``` + +#### Options +|Flag|Description| +|---|---| +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + + +### `status` + +Check the current status of a Pulsar Source. + +#### Usage + +```bash + +$ pulsar-admin sources status options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The source ID.
    If `instance-id` is not provided, Pulasr gets status of all instances.| +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + +### `list` + +List all running Pulsar IO source connectors. + +#### Usage + +```bash + +$ pulsar-admin sources list options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + + +### `stop` + +Stop a source instance. + +#### Usage + +```bash + +$ pulsar-admin sources stop options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The source instanceID.
    If `instance-id` is not provided, Pulsar stops all instances.| +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + +### `start` + +Start a source instance. + +#### Usage + +```bash + +$ pulsar-admin sources start options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The source instanceID.
    If `instance-id` is not provided, Pulsar starts all instances.| +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + + +### `restart` + +Restart a source instance. + +#### Usage + +```bash + +$ pulsar-admin sources restart options + +``` + +#### Options +|Flag|Description| +|---|---| +|`--instance-id`|The source instanceID.
    If `instance-id` is not provided, Pulsar restarts all instances. +|`--name`|The source's name.| +|`--namespace`|The source's namespace.| +|`--tenant`|The source's tenant.| + + +### `localrun` + +Run a Pulsar IO source connector locally rather than deploying it to the Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sources localrun options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the NAR archive for the Source.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--broker-service-url` | The URL for the Pulsar broker. +|`--classname`|The source's class name if `archive` is file-url-path (file://). +| `--client-auth-params` | Client authentication parameter. +| `--client-auth-plugin` | Client authentication plugin using which function-process can connect to broker. +|`--cpu`|The CPU (in cores) that needs to be allocated per source instance (applicable only to the Docker runtime).| +|`--deserialization-classname`|The SerDe classname for the source. +|`--destination-topic-name`|The Pulsar topic to which data is sent. +|`--disk`|The disk (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime).| +|`--hostname-verification-enabled`|Enable hostname verification.
    **Default value: false**. +|`--name`|The source’s name.| +|`--namespace`|The source’s namespace.| +|`--parallelism`|The source’s parallelism factor, that is, the number of source instances to run).| +|`--processing-guarantees` | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from external system and writes messages to a Pulsar topic. The `--processing-guarantees` is used to ensure the processing guarantees for writing messages to the Pulsar topic.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +|`--ram`|The RAM (in bytes) that needs to be allocated per source instance (applicable only to the Docker runtime).| +| `-st`, `--schema-type` | The schema type.
    Either a builtin schema (for example, AVRO and JSON) or custom schema class name to be used to encode messages emitted from source. +|`--source-config`|Source config key/values. +|`--source-config-file`|The path to a YAML config file specifying the source’s configuration. +|`--source-type`|The source's connector provider. +|`--tenant`|The source’s tenant. +|`--tls-allow-insecure`|Allow insecure tls connection.
    **Default value: false**. +|`--tls-trust-cert-path`|The tls trust cert file path. +|`--use-tls`|Use tls connection.
    **Default value: false**. +|`--producer-config`| The custom producer configuration (as a JSON string). + +### `available-sources` + +Get the list of Pulsar IO connector sources supported by Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sources available-sources + +``` + +### `reload` + +Reload the available built-in connectors. + +#### Usage + +```bash + +$ pulsar-admin sources reload + +``` + +## `sinks` + +An interface for managing Pulsar IO sinks (egress data from Pulsar). + +```bash + +$ pulsar-admin sinks subcommands + +``` + +Subcommands are: + +* `create` + +* `update` + +* `delete` + +* `get` + +* `status` + +* `list` + +* `stop` + +* `start` + +* `restart` + +* `localrun` + +* `available-sinks` + +* `reload` + + +### `create` + +Submit a Pulsar IO sink connector to run in a Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sinks create options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the archive file for the sink.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--auto-ack` | Whether or not the framework will automatically acknowledge messages. +| `--classname` | The sink's class name if `archive` is file-url-path (file://). +| `--cpu` | The CPU (in cores) that needs to be allocated per sink instance (applicable only to Docker runtime). +| `--custom-schema-inputs` | The map of input topics to schema types or class names (as a JSON string). +| `--custom-serde-inputs` | The map of input topics to SerDe class names (as a JSON string). +| `--disk` | The disk (in bytes) that needs to be allocated per sink instance (applicable only to Docker runtime). +|`-i, --inputs` | The sink's input topic or topics (multiple topics can be specified as a comma-separated list). +|`--name` | The sink's name. +| `--namespace` | The sink's namespace. +| ` --parallelism` | The sink's parallelism factor, that is, the number of sink instances to run. +| `--processing-guarantees` | The processing guarantees (also known as delivery semantics) applied to the sink. The `--processing-guarantees` implementation in Pulsar also relies on sink implementation.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +| `--ram` | The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the process and Docker runtimes). +| `--retain-ordering` | Sink consumes and sinks messages in order. +| `--sink-config` | sink config key/values. +| `--sink-config-file` | The path to a YAML config file specifying the sink's configuration. +| `-t`, `--sink-type` | The sink's connector provider. The `sink-type` parameter of the currently built-in connectors is determined by the setting of the `name` parameter specified in the pulsar-io.yaml file. +| `--subs-name` | Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer. +| `--tenant` | The sink's tenant. +| `--timeout-ms` | The message timeout in milliseconds. +| `--topics-pattern` | TopicsPattern to consume from list of topics under a namespace that match the pattern.
    `--input` and `--topics-Pattern` are mutually exclusive.
    Add SerDe class name for a pattern in `--customSerdeInputs` (supported for java fun only). + +### `update` + +Update a Pulsar IO sink connector. + +#### Usage + +```bash + +$ pulsar-admin sinks update options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the archive file for the sink.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--auto-ack` | Whether or not the framework will automatically acknowledge messages. +| `--classname` | The sink's class name if `archive` is file-url-path (file://). +| `--cpu` | The CPU (in cores) that needs to be allocated per sink instance (applicable only to Docker runtime). +| `--custom-schema-inputs` | The map of input topics to schema types or class names (as a JSON string). +| `--custom-serde-inputs` | The map of input topics to SerDe class names (as a JSON string). +| `--disk` | The disk (in bytes) that needs to be allocated per sink instance (applicable only to Docker runtime). +|`-i, --inputs` | The sink's input topic or topics (multiple topics can be specified as a comma-separated list). +|`--name` | The sink's name. +| `--namespace` | The sink's namespace. +| ` --parallelism` | The sink's parallelism factor, that is, the number of sink instances to run. +| `--processing-guarantees` | The processing guarantees (also known as delivery semantics) applied to the sink. The `--processing-guarantees` implementation in Pulsar also relies on sink implementation.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +| `--ram` | The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the process and Docker runtimes). +| `--retain-ordering` | Sink consumes and sinks messages in order. +| `--sink-config` | sink config key/values. +| `--sink-config-file` | The path to a YAML config file specifying the sink's configuration. +| `-t`, `--sink-type` | The sink's connector provider. +| `--subs-name` | Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer. +| `--tenant` | The sink's tenant. +| `--timeout-ms` | The message timeout in milliseconds. +| `--topics-pattern` | TopicsPattern to consume from list of topics under a namespace that match the pattern.
    `--input` and `--topics-Pattern` are mutually exclusive.
    Add SerDe class name for a pattern in `--customSerdeInputs` (supported for java fun only). +| `--update-auth-data` | Whether or not to update the auth data.
    **Default value: false.** + +### `delete` + +Delete a Pulsar IO sink connector. + +#### Usage + +```bash + +$ pulsar-admin sinks delete options + +``` + +#### Option + +|Flag|Description| +|---|---| +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + +### `get` + +Get the information about a Pulsar IO sink connector. + +#### Usage + +```bash + +$ pulsar-admin sinks get options + +``` + +#### Options +|Flag|Description| +|---|---| +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + + +### `status` + +Check the current status of a Pulsar sink. + +#### Usage + +```bash + +$ pulsar-admin sinks status options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The sink ID.
    If `instance-id` is not provided, Pulasr gets status of all instances.| +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + + +### `list` + +List all running Pulsar IO sink connectors. + +#### Usage + +```bash + +$ pulsar-admin sinks list options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + + +### `stop` + +Stop a sink instance. + +#### Usage + +```bash + +$ pulsar-admin sinks stop options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The sink instanceID.
    If `instance-id` is not provided, Pulsar stops all instances.| +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + +### `start` + +Start a sink instance. + +#### Usage + +```bash + +$ pulsar-admin sinks start options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The sink instanceID.
    If `instance-id` is not provided, Pulsar starts all instances.| +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + + +### `restart` + +Restart a sink instance. + +#### Usage + +```bash + +$ pulsar-admin sinks restart options + +``` + +#### Options + +|Flag|Description| +|---|---| +|`--instance-id`|The sink instanceID.
    If `instance-id` is not provided, Pulsar restarts all instances. +|`--name`|The sink's name.| +|`--namespace`|The sink's namespace.| +|`--tenant`|The sink's tenant.| + + +### `localrun` + +Run a Pulsar IO sink connector locally rather than deploying it to the Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sinks localrun options + +``` + +#### Options + +|Flag|Description| +|----|---| +| `-a`, `--archive` | The path to the archive file for the sink.
    It also supports url-path (http/https/file [file protocol assumes that file already exists on worker host]) from which worker can download the package. +| `--auto-ack` | Whether or not the framework will automatically acknowledge messages. +| `--broker-service-url` | The URL for the Pulsar broker. +|`--classname`|The sink's class name if `archive` is file-url-path (file://). +| `--client-auth-params` | Client authentication parameter. +| `--client-auth-plugin` | Client authentication plugin using which function-process can connect to broker. +|`--cpu`|The CPU (in cores) that needs to be allocated per sink instance (applicable only to the Docker runtime). +| `--custom-schema-inputs` | The map of input topics to Schema types or class names (as a JSON string). +| `--max-redeliver-count` | Maximum number of times that a message is redelivered before being sent to the dead letter queue. +| `--dead-letter-topic` | Name of the dead letter topic where the failing messages are sent. +| `--custom-serde-inputs` | The map of input topics to SerDe class names (as a JSON string). +|`--disk`|The disk (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime).| +|`--hostname-verification-enabled`|Enable hostname verification.
    **Default value: false**. +| `-i`, `--inputs` | The sink's input topic or topics (multiple topics can be specified as a comma-separated list). +|`--name`|The sink’s name.| +|`--namespace`|The sink’s namespace.| +|`--parallelism`|The sink’s parallelism factor, that is, the number of sink instances to run).| +|`--processing-guarantees`|The processing guarantees (also known as delivery semantics) applied to the sink. The `--processing-guarantees` implementation in Pulsar also relies on sink implementation.
    The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. +|`--ram`|The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the Docker runtime).| +|`--retain-ordering` | Sink consumes and sinks messages in order. +|`--sink-config`|sink config key/values. +|`--sink-config-file`|The path to a YAML config file specifying the sink’s configuration. +|`--sink-type`|The sink's connector provider. +|`--subs-name` | Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer. +|`--tenant`|The sink’s tenant. +| `--timeout-ms` | The message timeout in milliseconds. +| `--negative-ack-redelivery-delay-ms` | The negatively-acknowledged message redelivery delay in milliseconds. | +|`--tls-allow-insecure`|Allow insecure tls connection.
    **Default value: false**. +|`--tls-trust-cert-path`|The tls trust cert file path. +| `--topics-pattern` | TopicsPattern to consume from list of topics under a namespace that match the pattern.
    `--input` and `--topics-Pattern` are mutually exclusive.
    Add SerDe class name for a pattern in `--customSerdeInputs` (supported for java fun only). +|`--use-tls`|Use tls connection.
    **Default value: false**. + +### `available-sinks` + +Get the list of Pulsar IO connector sinks supported by Pulsar cluster. + +#### Usage + +```bash + +$ pulsar-admin sinks available-sinks + +``` + +### `reload` + +Reload the available built-in connectors. + +#### Usage + +```bash + +$ pulsar-admin sinks reload + +``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md b/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md new file mode 100644 index 0000000000000..47ee183c6367b --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md @@ -0,0 +1,236 @@ +--- +id: io-connectors +title: Built-in connector +sidebar_label: Built-in connector +original_id: io-connectors +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +Pulsar distribution includes a set of common connectors that have been packaged and tested with the rest of Apache Pulsar. These connectors import and export data from some of the most commonly used data systems. + +Using any of these connectors is as easy as writing a simple connector and running the connector locally or submitting the connector to a Pulsar Functions cluster. + +## Source connector + +Pulsar has various source connectors, which are sorted alphabetically as below. + +### Canal + +* [Configuration](io-canal-source.md#configuration) + +* [Example](io-canal-source.md#usage) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) + + +### Debezium MySQL + +* [Configuration](io-debezium-source.md#configuration) + +* [Example](io-debezium-source.md#example-of-mysql) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java) + +### Debezium PostgreSQL + +* [Configuration](io-debezium-source.md#configuration) + +* [Example](io-debezium-source.md#example-of-postgresql) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java) + +### Debezium MongoDB + +* [Configuration](io-debezium-source.md#configuration) + +* [Example](io-debezium-source.md#example-of-mongodb) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mongodb/src/main/java/org/apache/pulsar/io/debezium/mongodb/DebeziumMongoDbSource.java) + +### DynamoDB + +* [Configuration](io-dynamodb-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/dynamodb/src/main/java/org/apache/pulsar/io/dynamodb/DynamoDBSource.java) + +### File + +* [Configuration](io-file-source.md#configuration) + +* [Example](io-file-source.md#usage) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSource.java) + +### Flume + +* [Configuration](io-flume-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/FlumeConnector.java) + +### Twitter firehose + +* [Configuration](io-twitter-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java) + +### Kafka + +* [Configuration](io-kafka-source.md#configuration) + +* [Example](io-kafka-source.md#usage) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java) + +### Kinesis + +* [Configuration](io-kinesis-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSource.java) + +### Netty + +* [Configuration](io-netty-source.md#configuration) + +* [Example of TCP](io-netty-source.md#tcp) + +* [Example of HTTP](io-netty-source.md#http) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java) + +### NSQ + +* [Configuration](io-nsq-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/nsq/src/main/java/org/apache/pulsar/io/nsq/NSQSource.java) + +### RabbitMQ + +* [Configuration](io-rabbitmq-source.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java) + +## Sink connector + +Pulsar has various sink connectors, which are sorted alphabetically as below. + +### Aerospike + +* [Configuration](io-aerospike-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java) + +### Cassandra + +* [Configuration](io-cassandra-sink.md#configuration) + +* [Example](io-cassandra-sink.md#usage) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java) + +### ElasticSearch + +* [Configuration](io-elasticsearch-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java) + +### Flume + +* [Configuration](io-flume-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/sink/StringSink.java) + +### HBase + +* [Configuration](io-hbase-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hbase/src/main/java/org/apache/pulsar/io/hbase/HbaseAbstractConfig.java) + +### HDFS2 + +* [Configuration](io-hdfs2-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/AbstractHdfsConnector.java) + +### HDFS3 + +* [Configuration](io-hdfs3-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs3/src/main/java/org/apache/pulsar/io/hdfs3/AbstractHdfsConnector.java) + +### InfluxDB + +* [Configuration](io-influxdb-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/influxdb/src/main/java/org/apache/pulsar/io/influxdb/InfluxDBGenericRecordSink.java) + +### JDBC ClickHouse + +* [Configuration](io-jdbc-sink.md#configuration) + +* [Example](io-jdbc-sink.md#example-for-clickhouse) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/clickhouse/src/main/java/org/apache/pulsar/io/jdbc/ClickHouseJdbcAutoSchemaSink.java) + +### JDBC MariaDB + +* [Configuration](io-jdbc-sink.md#configuration) + +* [Example](io-jdbc-sink.md#example-for-mariadb) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/mariadb/src/main/java/org/apache/pulsar/io/jdbc/MariadbJdbcAutoSchemaSink.java) + +### JDBC PostgreSQL + +* [Configuration](io-jdbc-sink.md#configuration) + +* [Example](io-jdbc-sink.md#example-for-postgresql) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/postgres/src/main/java/org/apache/pulsar/io/jdbc/PostgresJdbcAutoSchemaSink.java) + +### JDBC SQLite + +* [Configuration](io-jdbc-sink.md#configuration) + +* [Example](io-jdbc-sink.md#example-for-sqlite) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/sqlite/src/main/java/org/apache/pulsar/io/jdbc/SqliteJdbcAutoSchemaSink.java) + +### Kafka + +* [Configuration](io-kafka-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java) + +### Kinesis + +* [Configuration](io-kinesis-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java) + +### MongoDB + +* [Configuration](io-mongo-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/mongo/src/main/java/org/apache/pulsar/io/mongodb/MongoSink.java) + +### RabbitMQ + +* [Configuration](io-rabbitmq-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSink.java) + +### Redis + +* [Configuration](io-redis-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/redis/src/main/java/org/apache/pulsar/io/redis/RedisAbstractConfig.java) + +### Solr + +* [Configuration](io-solr-sink.md#configuration) + +* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrSinkConfig.java) + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-debug.md b/site2/website-next/versioned_docs/version-2.7.2/io-debug.md new file mode 100644 index 0000000000000..fada2555acf71 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-debug.md @@ -0,0 +1,398 @@ +--- +id: io-debug +title: How to debug Pulsar connectors +sidebar_label: Debug +original_id: io-debug +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to debug connectors in localrun or cluster mode and gives a debugging checklist. +To better demonstrate how to debug Pulsar connectors, here takes a Mongo sink connector as an example. + +**Deploy a Mongo sink environment** +1. Start a Mongo service. + + ```bash + + docker pull mongo:4 + docker run -d -p 27017:27017 --name pulsar-mongo -v $PWD/data:/data/db mongo:4 + + ``` +2. Create a DB and a collection. + + ```bash + + docker exec -it pulsar-mongo /bin/bash + mongo + > use pulsar + > db.createCollection('messages') + > exit + + ``` +3. Start Pulsar standalone. + + ```bash + + docker pull apachepulsar/pulsar:2.4.0 + docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --link pulsar-mongo --name pulsar-mongo-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone + + ``` +4. Configure the Mongo sink with the `mongo-sink-config.yaml` file. + + ```bash + + configs: + mongoUri: "mongodb://pulsar-mongo:27017" + database: "pulsar" + collection: "messages" + batchSize: 2 + batchTimeMs: 500 + + ``` + ```bash + + docker cp mongo-sink-config.yaml pulsar-mongo-standalone:/pulsar/ + + ``` +5. Download the Mongo sink nar package. + + ```bash + + docker exec -it pulsar-mongo-standalone /bin/bash + curl -O http://apache.01link.hk/pulsar/pulsar-2.4.0/connectors/pulsar-io-mongo-2.4.0.nar + + ``` +## Debug in localrun mode +Start the Mongo sink in localrun mode using the `localrun` command. +:::tip + + +For more information about the `localrun` command, see [`localrun`](reference-connector-admin.md/#localrun-1). + +::: + +```bash + +./bin/pulsar-admin sinks localrun \ +--archive pulsar-io-mongo-2.4.0.nar \ +--tenant public --namespace default \ +--inputs test-mongo \ +--name pulsar-mongo-sink \ +--sink-config-file mongo-sink-config.yaml \ +--parallelism 1 + +``` +### Use connector log +Use one of the following methods to get a connector log in localrun mode: +* After executing the `localrun` command, the **log is automatically printed on the console**. +* The log is located at: + + + ```bash + + logs/functions/tenant/namespace/function-name/function-name-instance-id.log + + ``` + + **Example** + + The path of the Mongo sink connector is: + + ```bash + + logs/functions/public/default/pulsar-mongo-sink/pulsar-mongo-sink-0.log + + ``` +To clearly explain the log information, here breaks down the large block of information into small blocks and add descriptions for each block. +* This piece of log information shows the storage path of the nar package after decompression. + + ``` + 08:21:54.132 [main] INFO org.apache.pulsar.common.nar.NarClassLoader - Created class loader with paths: [file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/, file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/, + + ``` + +:::tip + +If `class cannot be found` exception is thrown, check whether the nar file is decompressed in the folder `file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/` or not. + +::: +* This piece of log information illustrates the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, resources, and so on, which can be used to **check whether the Mongo sink connector is configured correctly or not**. + + ```bash + + 08:21:55.390 [main] INFO org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=853d60a1-0c48-44d5-9a5c-6917386476b2, functionVersion=c2ce1458-b69e-4175-88c0-a0a856a2be8c, functionDetails=tenant: "public" + namespace: "default" + name: "pulsar-mongo-sink" + className: "org.apache.pulsar.functions.api.utils.IdentityFunction" + autoAck: true + parallelism: 1 + source { + typeClassName: "[B" + inputSpecs { + key: "test-mongo" + value { + } + } + cleanupSubscription: true + } + sink { + className: "org.apache.pulsar.io.mongodb.MongoSink" + configs: "{\"mongoUri\":\"mongodb://pulsar-mongo:27017\",\"database\":\"pulsar\",\"collection\":\"messages\",\"batchSize\":2,\"batchTimeMs\":500}" + typeClassName: "[B" + } + resources { + cpu: 1.0 + ram: 1073741824 + disk: 10737418240 + } + componentType: SINK + , maxBufferedTuples=1024, functionAuthenticationSpec=null, port=38459, clusterName=local) + + ``` +* This piece of log information demonstrates the status of the connections to Mongo and configuration information. + + ```bash + + 08:21:56.231 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to pulsar-mongo:27017 + 08:21:56.326 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=pulsar-mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 0]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=89058800} + + ``` +* This piece of log information explains the configuration of consumers and clients, including the topic name, subscription name, subscription type, and so on. + + ```bash + + 08:21:56.719 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: { + "topicNames" : [ "test-mongo" ], + "topicsPattern" : null, + "subscriptionName" : "public/default/pulsar-mongo-sink", + "subscriptionType" : "Shared", + "receiverQueueSize" : 1000, + "acknowledgementsGroupTimeMicros" : 100000, + "negativeAckRedeliveryDelayMicros" : 60000000, + "maxTotalReceiverQueueSizeAcrossPartitions" : 50000, + "consumerName" : null, + "ackTimeoutMillis" : 0, + "tickDurationMillis" : 1000, + "priorityLevel" : 0, + "cryptoFailureAction" : "CONSUME", + "properties" : { + "application" : "pulsar-sink", + "id" : "public/default/pulsar-mongo-sink", + "instance_id" : "0" + }, + "readCompacted" : false, + "subscriptionInitialPosition" : "Latest", + "patternAutoDiscoveryPeriod" : 1, + "regexSubscriptionMode" : "PersistentOnly", + "deadLetterPolicy" : null, + "autoUpdatePartitions" : true, + "replicateSubscriptionState" : false, + "resetIncludeHead" : false + } + 08:21:56.726 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: { + "serviceUrl" : "pulsar://localhost:6650", + "authPluginClassName" : null, + "authParams" : null, + "operationTimeoutMs" : 30000, + "statsIntervalSeconds" : 60, + "numIoThreads" : 1, + "numListenerThreads" : 1, + "connectionsPerBroker" : 1, + "useTcpNoDelay" : true, + "useTls" : false, + "tlsTrustCertsFilePath" : null, + "tlsAllowInsecureConnection" : false, + "tlsHostnameVerificationEnable" : false, + "concurrentLookupRequest" : 5000, + "maxLookupRequest" : 50000, + "maxNumberOfRejectedRequestPerConnection" : 50, + "keepAliveIntervalSeconds" : 30, + "connectionTimeoutMs" : 10000, + "requestTimeoutMs" : 60000, + "defaultBackoffIntervalNanos" : 100000000, + "maxBackoffIntervalNanos" : 30000000000 + } + + ``` +## Debug in cluster mode +You can use the following methods to debug a connector in cluster mode: +* [Use connector log](#use-connector-log) +* [Use admin CLI](#use-admin-cli) +### Use connector log +In cluster mode, multiple connectors can run on a worker. To find the log path of a specified connector, use the `workerId` to locate the connector log. +### Use admin CLI +Pulsar admin CLI helps you debug Pulsar connectors with the following subcommands: +* [`get`](#get) + +* [`status`](#status) +* [`topics stats`](#topics-stats) + +**Create a Mongo sink** + +```bash + +./bin/pulsar-admin sinks create \ +--archive pulsar-io-mongo-2.4.0.nar \ +--tenant public \ +--namespace default \ +--inputs test-mongo \ +--name pulsar-mongo-sink \ +--sink-config-file mongo-sink-config.yaml \ +--parallelism 1 + +``` +### `get` +Use the `get` command to get the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, and so on. + +```bash + +./bin/pulsar-admin sinks get --tenant public --namespace default --name pulsar-mongo-sink +{ + "tenant": "public", + "namespace": "default", + "name": "pulsar-mongo-sink", + "className": "org.apache.pulsar.io.mongodb.MongoSink", + "inputSpecs": { + "test-mongo": { + "isRegexPattern": false + } + }, + "configs": { + "mongoUri": "mongodb://pulsar-mongo:27017", + "database": "pulsar", + "collection": "messages", + "batchSize": 2.0, + "batchTimeMs": 500.0 + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true +} + +``` + +:::tip + + +For more information about the `get` command, see [`get`](reference-connector-admin.md/#get-1). + +::: +### `status` +Use the `status` command to get the current status about the Mongo sink connector, such as the number of instance, the number of running instance, instanceId, workerId and so on. + +```bash + +./bin/pulsar-admin sinks status +--tenant public \ +--namespace default \ +--name pulsar-mongo-sink +{ +"numInstances" : 1, +"numRunning" : 1, +"instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 0, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 0, + "lastReceivedTime" : 0, + "workerId" : "c-standalone-fw-5d202832fd18-8080" + } +} ] +} + +``` +:::tip + + +For more information about the `status` command, see [`status`](reference-connector-admin.md/#stauts-1). + +If there are multiple connectors running on a worker, `workerId` can locate the worker on which the specified connector is running. + +::: +### `topics stats` +Use the `topics stats` command to get the stats for a topic and its connected producer and consumer, such as whether the topic has received messages or not, whether there is a backlog of messages or not, the available permits and other key information. All rates are computed over a 1-minute window and are relative to the last completed 1-minute period. + +```bash + +./bin/pulsar-admin topics stats test-mongo +{ + "msgRateIn" : 0.0, + "msgThroughputIn" : 0.0, + "msgRateOut" : 0.0, + "msgThroughputOut" : 0.0, + "averageMsgSize" : 0.0, + "storageSize" : 1, + "publishers" : [ ], + "subscriptions" : { + "public/default/pulsar-mongo-sink" : { + "msgRateOut" : 0.0, + "msgThroughputOut" : 0.0, + "msgRateRedeliver" : 0.0, + "msgBacklog" : 0, + "blockedSubscriptionOnUnackedMsgs" : false, + "msgDelayed" : 0, + "unackedMessages" : 0, + "type" : "Shared", + "msgRateExpired" : 0.0, + "consumers" : [ { + "msgRateOut" : 0.0, + "msgThroughputOut" : 0.0, + "msgRateRedeliver" : 0.0, + "consumerName" : "dffdd", + "availablePermits" : 999, + "unackedMessages" : 0, + "blockedConsumerOnUnackedMsgs" : false, + "metadata" : { + "instance_id" : "0", + "application" : "pulsar-sink", + "id" : "public/default/pulsar-mongo-sink" + }, + "connectedSince" : "2019-08-26T08:48:07.582Z", + "clientVersion" : "2.4.0", + "address" : "/172.17.0.3:57790" + } ], + "isReplicated" : false + } + }, + "replication" : { }, + "deduplicationStatus" : "Disabled" +} + +``` +:::tip + + +For more information about the `topic stats` command, see [`topic stats`](http://pulsar.apache.org/docs/en/pulsar-admin/#stats-1). + +::: +## Checklist +This checklist indicates the major areas to check when you debug connectors. It is a reminder of what to look for to ensure a thorough review and an evaluation tool to get the status of connectors. +* Does Pulsar start successfully? + +* Does the external service run normally? + +* Is the nar package complete? + +* Is the connector configuration file correct? + +* In localrun mode, run a connector and check the printed information (connector log) on the console. + +* In cluster mode： + + * Use the `get` command to get the basic information. + + * Use the `status` command to get the current status. + * Use the `topics stats` command to get the stats for a specified topic and its connected producers and consumers. + + * Check the connector log. +* Enter into the external system and verify the result. diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-develop.md b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md new file mode 100644 index 0000000000000..df45d9324edd3 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md @@ -0,0 +1,270 @@ +--- +id: io-develop +title: How to develop Pulsar connectors +sidebar_label: Develop +original_id: io-develop +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This guide describes how to develop Pulsar connectors to move data +between Pulsar and other systems. + +Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +a Pulsar connector is similar to creating a Pulsar function. + +Pulsar connectors come in two types: + +| Type | Description | Example +|---|---|--- +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. + +## Develop + +You can develop Pulsar source connectors and sink connectors. + +### Source + +Developing a source connector is to implement the {@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} +interface, which means you need to implement the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method and the {@inject: github:read:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method. + +1. Implement the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method. + + ```java + + /** + * Open connector with configuration + * + * @param config initialization config + * @param sourceContext + * @throws Exception IO type exceptions when opening a connector + */ + void open(final Map config, SourceContext sourceContext) throws Exception; + + ``` + + This method is called when the source connector is initialized. + + In this method, you can retrieve all connector specific settings through the passed-in `config` parameter and initialize all necessary resources. + + For example, a Kafka connector can create a Kafka client in this `open` method. + + Besides, Pulsar runtime also provides a `SourceContext` for the + connector to access runtime resources for tasks like collecting metrics. The implementation can save the `SourceContext` for future use. + +2. Implement the {@inject: github:read:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method. + + ```java + + /** + * Reads the next message from source. + * If source does not have any new messages, this call should block. + * @return next message from source. The return result should never be null + * @throws Exception + */ + Record read() throws Exception; + + ``` + + If nothing to return, the implementation should be blocking rather than returning `null`. + + The returned {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should encapsulate the following information, which is needed by Pulsar IO runtime. + + * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following variables: + + |Variable|Required|Description + |---|---|--- + `TopicName`|No|Pulsar topic name from which the record is originated from. + `Key`|No| Messages can optionally be tagged with keys.

    For more information, see [Routing modes](concepts-messaging.md#routing-modes).| + `Value`|Yes|Actual data of the record. + `EventTime`|No|Event time of the record from the source. + `PartitionId`|No| If the record is originated from a partitioned source, it returns its `PartitionId`.

    `PartitionId` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. + `RecordSequence`|No|If the record is originated from a sequential source, it returns its `RecordSequence`.

    `RecordSequence` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. + `Properties` |No| If the record carries user-defined properties, it returns those properties. + `DestinationTopic`|No|Topic to which message should be written. + `Message`|No|A class which carries data sent by users.

    For more information, see [Message.java](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java).| + + * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following methods: + + Method|Description + |---|--- + `ack` |Acknowledge that the record is fully processed. + `fail`|Indicate that the record fails to be processed. + +:::tip + + +For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. + +::: + +### Sink + +Developing a sink connector **is similar to** developing a source connector, that is, you need to implement the {@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} interface, which means implementing the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method and the {@inject: github:write:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method. + +1. Implement the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method. + + ```java + + /** + * Open connector with configuration + * + * @param config initialization config + * @param sinkContext + * @throws Exception IO type exceptions when opening a connector + */ + void open(final Map config, SinkContext sinkContext) throws Exception; + + ``` + +2. Implement the {@inject: github:write:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method. + + ```java + + /** + * Write a message to Sink + * @param record record to write to sink + * @throws Exception + */ + void write(Record record) throws Exception; + + ``` + + During the implementation, you can decide how to write the `Value` and + the `Key` to the actual source, and leverage all the provided information such as + `PartitionId` and `RecordSequence` to achieve different processing guarantees. + + You also need to ack records (if messages are sent successfully) or fail records (if messages fail to send). + +## Test + +Testing connectors can be challenging because Pulsar IO connectors interact with two systems +that may be difficult to mock—Pulsar and the system to which the connector is connecting. + +It is +recommended writing special tests to test the connector functionalities as below +while mocking the external service. + +### Unit test + +You can create unit tests for your connector. + +### Integration test + +Once you have written sufficient unit tests, you can add +separate integration tests to verify end-to-end functionality. + +Pulsar uses [testcontainers](https://www.testcontainers.org/) **for all integration tests**. + +:::tip + + +For more information about **how to create integration tests for Pulsar connectors**, see {@inject: github:IntegrationTests:/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io}. + +::: + +## Package + +Once you've developed and tested your connector, you need to package it so that it can be submitted +to a [Pulsar Functions](functions-overview) cluster. + +There are two methods to +work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). + +:::note + + +If you plan to package and distribute your connector for others to use, you are obligated to + +::: +license and copyright your own code properly. Remember to add the license and copyright to +all libraries your code uses and to your distribution. +> +> If you use the [NAR](#nar) method, the NAR plugin +automatically creates a `DEPENDENCIES` file in the generated NAR package, including the proper +licensing and copyrights of all libraries of your connector. + +### NAR + +**NAR** stands for NiFi Archive, which is a custom packaging mechanism used by Apache NiFi, to provide +a bit of Java ClassLoader isolation. + +:::tip + + +For more information about **how NAR works**, see [here](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd). + +::: + +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). + +The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). + +Include this [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin) in your maven project for your connector as below. + +```xml + + + + org.apache.nifi + nifi-nar-maven-plugin + 1.2.0 + + + +``` + +You must also create a `resources/META-INF/services/pulsar-io.yaml` file with the following contents: + +```yaml + +name: connector name +description: connector description +sourceClass: fully qualified class name (only if source connector) +sinkClass: fully qualified class name (only if sink connector) + +``` + +For Gradle users, there is a [Gradle Nar plugin available on the Gradle Plugin Portal](https://plugins.gradle.org/plugin/io.github.lhotari.gradle-nar-plugin). + +:::tip + + +For more information about an **how to use NAR for Pulsar connectors**, see {@inject: github:TwitterFirehose:/pulsar-io/twitter/pom.xml}. + +::: + +### Uber JAR + +An alternative approach is to create an **uber JAR** that contains all of the connector's JAR files +and other resource files. No directory internal structure is necessary. + +You can use [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html) to create a uber JAR as below: + +```xml + + + org.apache.maven.plugins + maven-shade-plugin + 3.1.1 + + + package + + shade + + + + + *:* + + + + + + + +``` diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-overview.md b/site2/website-next/versioned_docs/version-2.7.2/io-overview.md new file mode 100644 index 0000000000000..adb499818b0c9 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-overview.md @@ -0,0 +1,176 @@ +--- +id: io-overview +title: Pulsar connector overview +sidebar_label: Overview +original_id: io-overview +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +Messaging systems are most powerful when you can easily use them with external systems like databases and other messaging systems. + +**Pulsar IO connectors** enable you to easily create, deploy, and manage connectors that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. + + +## Concept + +Pulsar IO connectors come in two types: **source** and **sink**. + +This diagram illustrates the relationship between source, Pulsar, and sink: + +![Pulsar IO diagram](/assets/pulsar-io.png "Pulsar IO connectors (sources and sinks)") + + +### Source + +> Sources **feed data from external systems into Pulsar**. + +Common sources include other messaging systems and firehose-style data pipeline APIs. + +For the complete list of Pulsar built-in source connectors, see [source connector](io-connectors.md#source-connector). + +### Sink + +> Sinks **feed data from Pulsar into external systems**. + +Common sinks include other messaging systems and SQL and NoSQL databases. + +For the complete list of Pulsar built-in sink connectors, see [sink connector](io-connectors.md#sink-connector). + +## Processing guarantee + +Processing guarantees are used to handle errors when writing messages to Pulsar topics. + +> Pulsar connectors and Functions use the **same** processing guarantees as below. + +Delivery semantic | Description +:------------------|:------- +`at-most-once` | Each message sent to a connector is to be **processed once** or **not to be processed**. +`at-least-once` | Each message sent to a connector is to be **processed once** or **more than once**. +`effectively-once` | Each message sent to a connector has **one output associated** with it. + +> Processing guarantees for connectors not just rely on Pulsar guarantee but also **relate to external systems**, that is, **the implementation of source and sink**. + +* Source: Pulsar ensures that writing messages to Pulsar topics respects to the processing guarantees. It is within Pulsar's control. + +* Sink: the processing guarantees rely on the sink implementation. If the sink implementation does not handle retries in an idempotent way, the sink does not respect to the processing guarantees. + +### Set + +When creating a connector, you can set the processing guarantee with the following semantics: + +* ATLEAST_ONCE + +* ATMOST_ONCE + +* EFFECTIVELY_ONCE + +> If `--processing-guarantees` is not specified when creating a connector, the default semantic is `ATLEAST_ONCE`. + +Here takes **Admin CLI** as an example. For more information about **REST API** or **JAVA Admin API**, see [here](io-use.md#create). + + + + + +```bash + +$ bin/pulsar-admin sources create \ + --processing-guarantees ATMOST_ONCE \ + # Other source configs + +``` + +For more information about the options of `pulsar-admin sources create`, see [here](reference-connector-admin.md#create). + + + + +```bash + +$ bin/pulsar-admin sinks create \ + --processing-guarantees EFFECTIVELY_ONCE \ + # Other sink configs + +``` + +For more information about the options of `pulsar-admin sinks create`, see [here](reference-connector-admin.md#create-1). + + + + + +### Update + +After creating a connector, you can update the processing guarantee with the following semantics: + +* ATLEAST_ONCE + +* ATMOST_ONCE + +* EFFECTIVELY_ONCE + +Here takes **Admin CLI** as an example. For more information about **REST API** or **JAVA Admin API**, see [here](io-use.md#create). + + + + + +```bash + +$ bin/pulsar-admin sources update \ + --processing-guarantees EFFECTIVELY_ONCE \ + # Other source configs + +``` + +For more information about the options of `pulsar-admin sources update`, see [here](reference-connector-admin.md#update). + + + + +```bash + +$ bin/pulsar-admin sinks update \ + --processing-guarantees ATMOST_ONCE \ + # Other sink configs + +``` + +For more information about the options of `pulsar-admin sinks update`, see [here](reference-connector-admin.md#update-1). + + + + + + +## Work with connector + +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. + +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md new file mode 100644 index 0000000000000..6df4de8a66903 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md @@ -0,0 +1,983 @@ +--- +id: io-quickstart +title: How to connect Pulsar to database +sidebar_label: Get started +original_id: io-quickstart +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. + +It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. + +At the end of this tutorial, you are able to: + +- [Connect Pulsar to Cassandra](#Connect-Pulsar-to-Cassandra) + +- [Connect Pulsar to PostgreSQL](#Connect-Pulsar-to-PostgreSQL) + +:::tip + + +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. + +* All the instructions are assumed to run at the root directory of a Pulsar binary distribution. + +::: + +## Install Pulsar and built-in connector + +Before connecting Pulsar to a database, you need to install Pulsar and the desired built-in connector. + +For more information about **how to install a standalone Pulsar and built-in connectors**, see [here](getting-started-standalone.md/#installing-pulsar). + +## Start Pulsar standalone + +1. Start Pulsar locally. + + ```bash + + bin/pulsar standalone + + ``` + + All the components of a Pulsar service are start in order. + + You can curl those pulsar service endpoints to make sure Pulsar service is up running correctly. + +2. Check Pulsar binary protocol port. + + ```bash + + telnet localhost 6650 + + ``` + +3. Check Pulsar Function cluster. + + ```bash + + curl -s http://localhost:8080/admin/v2/worker/cluster + + ``` + + **Example output** + + ```json + + [{"workerId":"c-standalone-fw-localhost-6750","workerHostname":"localhost","port":6750}] + + ``` + +4. Make sure a public tenant and a default namespace exist. + + ```bash + + curl -s http://localhost:8080/admin/v2/namespaces/public + + ``` + + **Example output** + + ```json + + ["public/default","public/functions"] + + ``` + +5. All built-in connectors should be listed as available. + + ```bash + + curl -s http://localhost:8080/admin/v2/functions/connectors + + ``` + + **Example output** + + ```json + + [{"name":"aerospike","description":"Aerospike database sink","sinkClass":"org.apache.pulsar.io.aerospike.AerospikeStringSink"},{"name":"cassandra","description":"Writes data into Cassandra","sinkClass":"org.apache.pulsar.io.cassandra.CassandraStringSink"},{"name":"kafka","description":"Kafka source and sink connector","sourceClass":"org.apache.pulsar.io.kafka.KafkaStringSource","sinkClass":"org.apache.pulsar.io.kafka.KafkaBytesSink"},{"name":"kinesis","description":"Kinesis sink connector","sinkClass":"org.apache.pulsar.io.kinesis.KinesisSink"},{"name":"rabbitmq","description":"RabbitMQ source connector","sourceClass":"org.apache.pulsar.io.rabbitmq.RabbitMQSource"},{"name":"twitter","description":"Ingest data from Twitter firehose","sourceClass":"org.apache.pulsar.io.twitter.TwitterFireHose"}] + + ``` + + If an error occurs when starting Pulsar service, you may see an exception at the terminal running `pulsar/standalone`, + or you can navigate to the `logs` directory under the Pulsar directory to view the logs. + +## Connect Pulsar to Cassandra + +This section demonstrates how to connect Pulsar to Cassandra. + +:::tip + + +* Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). + +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). + +::: + +### Setup a Cassandra cluster + +This example uses `cassandra` Docker image to start a single-node Cassandra cluster in Docker. + +1. Start a Cassandra cluster. + +```bash + +docker run -d --rm --name=cassandra -p 9042:9042 cassandra + +``` + +:::note + + Before moving to the next steps, make sure the Cassandra cluster is running. + + ::: + +2. Make sure the Docker process is running. + + ```bash + + docker ps + + ``` + +3. Check the Cassandra logs to make sure the Cassandra process is running as expected. + + ```bash + + docker logs cassandra + + ``` + +4. Check the status of the Cassandra cluster. + + ```bash + + docker exec cassandra nodetool status + + ``` + + **Example output** + + ``` + Datacenter: datacenter1 + ======================= + Status=Up/Down + |/ State=Normal/Leaving/Joining/Moving + -- Address Load Tokens Owns (effective) Host ID Rack + UN 172.17.0.2 103.67 KiB 256 100.0% af0e4b2f-84e0-4f0b-bb14-bd5f9070ff26 rack1 + + ``` + +5. Use `cqlsh` to connect to the Cassandra cluster. + + ```bash + + $ docker exec -ti cassandra cqlsh localhost + Connected to Test Cluster at localhost:9042. + [cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4] + Use HELP for help. + cqlsh> + + ``` + +6. Create a keyspace `pulsar_test_keyspace`. + + ```bash + + cqlsh> CREATE KEYSPACE pulsar_test_keyspace WITH replication = {'class':'SimpleStrategy', 'replication_factor':1}; + + ``` + +7. Create a table `pulsar_test_table`. + + ```bash + + cqlsh> USE pulsar_test_keyspace; + cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY KEY, col text); + + ``` + +### Configure a Cassandra sink + +Now that we have a Cassandra cluster running locally. + +In this section, you need to configure a Cassandra sink connector. + +To run a Cassandra sink connector, you need to prepare a configuration file including the information that Pulsar connector runtime needs to know. + +For example, how Pulsar connector can find the Cassandra cluster, what is the keyspace and the table that Pulsar connector uses for writing Pulsar messages to, and so on. + +You can create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "roots": "localhost:9042", + "keyspace": "pulsar_test_keyspace", + "columnFamily": "pulsar_test_table", + "keyname": "key", + "columnName": "col" + } + + ``` + +* YAML + + + ```yaml + + configs: + roots: "localhost:9042" + keyspace: "pulsar_test_keyspace" + columnFamily: "pulsar_test_table" + keyname: "key" + columnName: "col" + + ``` + +For more information, see [Cassandra sink connector](io-cassandra-sink). + +### Create a Cassandra sink + +You can use the [Connector Admin CLI](io-cli) +to create a sink connector and perform other operations on them. + +Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. + +#### Note +> The `sink-type` parameter of the currently built-in connectors is determined by the setting of the `name` parameter specified in the pulsar-io.yaml file. + +```bash + +bin/pulsar-admin sinks create \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink \ + --sink-type cassandra \ + --sink-config-file examples/cassandra-sink.yml \ + --inputs test_cassandra + +``` + +Once the command is executed, Pulsar creates the sink connector _cassandra-test-sink_. + +This sink connector runs +as a Pulsar Function and writes the messages produced in the topic _test_cassandra_ to the Cassandra table _pulsar_test_table_. + +### Inspect a Cassandra sink + +You can use the [Connector Admin CLI](io-cli) +to monitor a connector and perform other operations on it. + +* Get the information of a Cassandra sink. + + ```bash + + bin/pulsar-admin sinks get \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink + + ``` + + **Example output** + + ```json + + { + "tenant": "public", + "namespace": "default", + "name": "cassandra-test-sink", + "className": "org.apache.pulsar.io.cassandra.CassandraStringSink", + "inputSpecs": { + "test_cassandra": { + "isRegexPattern": false + } + }, + "configs": { + "roots": "localhost:9042", + "keyspace": "pulsar_test_keyspace", + "columnFamily": "pulsar_test_table", + "keyname": "key", + "columnName": "col" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true, + "archive": "builtin://cassandra" + } + + ``` + +* Check the status of a Cassandra sink. + + ```bash + + bin/pulsar-admin sinks status \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink + + ``` + + **Example output** + + ```json + + { + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 0, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 0, + "lastReceivedTime" : 0, + "workerId" : "c-standalone-fw-localhost-8080" + } + } ] + } + + ``` + +### Verify a Cassandra sink + +1. Produce some messages to the input topic of the Cassandra sink _test_cassandra_. + + ```bash + + for i in {0..9}; do bin/pulsar-client produce -m "key-$i" -n 1 test_cassandra; done + + ``` + +2. Inspect the status of the Cassandra sink _test_cassandra_. + + ```bash + + bin/pulsar-admin sinks status \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink + + ``` + + You can see 10 messages are processed by the Cassandra sink _test_cassandra_. + + **Example output** + + ```json + + { + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 10, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 10, + "lastReceivedTime" : 1551685489136, + "workerId" : "c-standalone-fw-localhost-8080" + } + } ] + } + + ``` + +3. Use `cqlsh` to connect to the Cassandra cluster. + + ```bash + + docker exec -ti cassandra cqlsh localhost + + ``` + +4. Check the data of the Cassandra table _pulsar_test_table_. + + ```bash + + cqlsh> use pulsar_test_keyspace; + cqlsh:pulsar_test_keyspace> select * from pulsar_test_table; + + key | col + --------+-------- + key-5 | key-5 + key-0 | key-0 + key-9 | key-9 + key-2 | key-2 + key-1 | key-1 + key-3 | key-3 + key-6 | key-6 + key-7 | key-7 + key-4 | key-4 + key-8 | key-8 + + ``` + +### Delete a Cassandra Sink + +You can use the [Connector Admin CLI](io-cli) +to delete a connector and perform other operations on it. + +```bash + +bin/pulsar-admin sinks delete \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink + +``` + +## Connect Pulsar to PostgreSQL + +This section demonstrates how to connect Pulsar to PostgreSQL. + +:::tip + + +* Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). + +* The JDBC sink connector pulls messages from Pulsar topics + +::: +and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. +>For more information, see [JDBC sink connector](io-jdbc-sink). + + +### Setup a PostgreSQL cluster + +This example uses the PostgreSQL 12 docker image to start a single-node PostgreSQL cluster in Docker. + +1. Pull the PostgreSQL 12 image from Docker. + + ```bash + + $ docker pull postgres:12 + + ``` + +2. Start PostgreSQL. + + ```bash + + $ docker run -d -it --rm \ + --name pulsar-postgres \ + -p 5432:5432 \ + -e POSTGRES_PASSWORD=password \ + -e POSTGRES_USER=postgres \ + postgres:12 + + ``` + + #### Tip + + Flag | Description | This example + ---|---|---| + `-d` | To start a container in detached mode. | / + `-it` | Keep STDIN open even if not attached and allocate a terminal. | / + `--rm` | Remove the container automatically when it exits. | / + `-name` | Assign a name to the container. | This example specifies _pulsar-postgres_ for the container. + `-p` | Publish the port of the container to the host. | This example publishes the port _5432_ of the container to the host. + `-e` | Set environment variables. | This example sets the following variables:
    - The password for the user is _password_.
    - The name for the user is _postgres_. + + :::tip + + For more information about Docker commands, see [Docker CLI](https://docs.docker.com/engine/reference/commandline/run/). + + ::: + +3. Check if PostgreSQL has been started successfully. + + ```bash + + $ docker logs -f pulsar-postgres + + ``` + + PostgreSQL has been started successfully if the following message appears. + + ```text + + 2020-05-11 20:09:24.492 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit + 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 + 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv6 address "::", port 5432 + 2020-05-11 20:09:24.499 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" + 2020-05-11 20:09:24.523 UTC [55] LOG: database system was shut down at 2020-05-11 20:09:24 UTC + 2020-05-11 20:09:24.533 UTC [1] LOG: database system is ready to accept connections + + ``` + +4. Access to PostgreSQL. + + ```bash + + $ docker exec -it pulsar-postgres /bin/bash + + ``` + +5. Create a PostgreSQL table _pulsar_postgres_jdbc_sink_. + + ```bash + + $ psql -U postgres postgres + + postgres=# create table if not exists pulsar_postgres_jdbc_sink + ( + id serial PRIMARY KEY, + name VARCHAR(255) NOT NULL + ); + + ``` + +### Configure a JDBC sink + +Now we have a PostgreSQL running locally. + +In this section, you need to configure a JDBC sink connector. + +1. Add a configuration file. + + To run a JDBC sink connector, you need to prepare a YAML configuration file including the information that Pulsar connector runtime needs to know. + + For example, how Pulsar connector can find the PostgreSQL cluster, what is the JDBC URL and the table that Pulsar connector uses for writing messages to. + + Create a _pulsar-postgres-jdbc-sink.yaml_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. + + ```yaml + + configs: + userName: "postgres" + password: "password" + jdbcUrl: "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink" + tableName: "pulsar_postgres_jdbc_sink" + + ``` + +2. Create a schema. + + Create a _avro-schema_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. + + ```json + + { + "type": "AVRO", + "schema": "{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}", + "properties": {} + } + + ``` + + :::tip + + For more information about AVRO, see [Apache Avro](https://avro.apache.org/docs/1.9.1/). + + ::: + +3. Upload a schema to a topic. + + This example uploads the _avro-schema_ schema to the _pulsar-postgres-jdbc-sink-topic_ topic. + + ```bash + + $ bin/pulsar-admin schemas upload pulsar-postgres-jdbc-sink-topic -f ./connectors/avro-schema + + ``` + +4. Check if the schema has been uploaded successfully. + + ```bash + + $ bin/pulsar-admin schemas get pulsar-postgres-jdbc-sink-topic + + ``` + + The schema has been uploaded successfully if the following message appears. + + ```json + + {"name":"pulsar-postgres-jdbc-sink-topic","schema":"{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}","type":"AVRO","properties":{}} + + ``` + +### Create a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to create a sink connector and perform other operations on it. + +This example creates a sink connector and specifies the desired information. + +```bash + +$ bin/pulsar-admin sinks create \ +--archive ./connectors/pulsar-io-jdbc-postgres-@pulsar:version@.nar \ +--inputs pulsar-postgres-jdbc-sink-topic \ +--name pulsar-postgres-jdbc-sink \ +--sink-config-file ./connectors/pulsar-postgres-jdbc-sink.yaml \ +--parallelism 1 + +``` + +Once the command is executed, Pulsar creates a sink connector _pulsar-postgres-jdbc-sink_. + +This sink connector runs as a Pulsar Function and writes the messages produced in the topic _pulsar-postgres-jdbc-sink-topic_ to the PostgreSQL table _pulsar_postgres_jdbc_sink_. + + #### Tip + + Flag | Description | This example + ---|---|---| + `--archive` | The path to the archive file for the sink. | _pulsar-io-jdbc-postgres-@pulsar:version@.nar_ | + `--inputs` | The input topic(s) of the sink.

    Multiple topics can be specified as a comma-separated list.|| + `--name` | The name of the sink. | _pulsar-postgres-jdbc-sink_ | + `--sink-config-file` | The path to a YAML config file specifying the configuration of the sink. | _pulsar-postgres-jdbc-sink.yaml_ | + `--parallelism` | The parallelism factor of the sink.

    For example, the number of sink instances to run. | _1_ | + +:::tip + + +For more information about `pulsar-admin sinks create options`, see [here](io-cli.md#sinks). + +::: + +The sink has been created successfully if the following message appears. + +```bash + +"Created successfully" + +``` + +### Inspect a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to monitor a connector and perform other operations on it. + +* List all running JDBC sink(s). + + ```bash + + $ bin/pulsar-admin sinks list \ + --tenant public \ + --namespace default + + ``` + +:::tip + + +For more information about `pulsar-admin sinks list options`, see [here](io-cli.md/#list-1). + +::: + + The result shows that only the _postgres-jdbc-sink_ sink is running. + + ```json + + [ + "pulsar-postgres-jdbc-sink" + ] + + ``` + +* Get the information of a JDBC sink. + + ```bash + + $ bin/pulsar-admin sinks get \ + --tenant public \ + --namespace default \ + --name pulsar-postgres-jdbc-sink + + ``` + +:::tip + + +For more information about `pulsar-admin sinks get options`, see [here](io-cli.md/#get-1). + +::: + + The result shows the information of the sink connector, including tenant, namespace, topic and so on. + + ```json + + { + "tenant": "public", + "namespace": "default", + "name": "pulsar-postgres-jdbc-sink", + "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink", + "inputSpecs": { + "pulsar-postgres-jdbc-sink-topic": { + "isRegexPattern": false + } + }, + "configs": { + "password": "password", + "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", + "userName": "postgres", + "tableName": "pulsar_postgres_jdbc_sink" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true + } + + ``` + +* Get the status of a JDBC sink + + ```bash + + $ bin/pulsar-admin sinks status \ + --tenant public \ + --namespace default \ + --name pulsar-postgres-jdbc-sink + + ``` + +:::tip + + +For more information about `pulsar-admin sinks status options`, see [here](io-cli.md/#status-1). + +::: + + The result shows the current status of sink connector, including the number of instance, running status, worker ID and so on. + + ```json + + { + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 0, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 0, + "lastReceivedTime" : 0, + "workerId" : "c-standalone-fw-192.168.2.52-8080" + } + } ] + } + + ``` + +### Stop a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to stop a connector and perform other operations on it. + +```bash + +$ bin/pulsar-admin sinks stop \ +--tenant public \ +--namespace default \ +--name pulsar-postgres-jdbc-sink + +``` + +:::tip + + +For more information about `pulsar-admin sinks stop options`, see [here](io-cli.md/#stop-1). + +::: + +The sink instance has been stopped successfully if the following message disappears. + +```bash + +"Stopped successfully" + +``` + +### Restart a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to restart a connector and perform other operations on it. + +```bash + +$ bin/pulsar-admin sinks restart \ +--tenant public \ +--namespace default \ +--name pulsar-postgres-jdbc-sink + +``` + +:::tip + + +For more information about `pulsar-admin sinks restart options`, see [here](io-cli.md/#restart-1). + +::: + +The sink instance has been started successfully if the following message disappears. + +```bash + +"Started successfully" + +``` + +:::tip + + +* Optionally, you can run a standalone sink connector using `pulsar-admin sinks localrun options`. + +Note that `pulsar-admin sinks localrun options` **runs a sink connector locally**, while `pulsar-admin sinks start options` **starts a sink connector in a cluster**. + +* For more information about `pulsar-admin sinks localrun options`, see [here](io-cli.md#localrun-1). + +::: + +### Update a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to update a connector and perform other operations on it. + +This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. + +```bash + +$ bin/pulsar-admin sinks update \ +--name pulsar-postgres-jdbc-sink \ +--parallelism 2 + +``` + +:::tip + + +For more information about `pulsar-admin sinks update options`, see [here](io-cli.md/#update-1). + +::: + +The sink connector has been updated successfully if the following message disappears. + +```bash + +"Updated successfully" + +``` + +This example double-checks the information. + +```bash + +$ bin/pulsar-admin sinks get \ +--tenant public \ +--namespace default \ +--name pulsar-postgres-jdbc-sink + +``` + +The result shows that the parallelism is 2. + +```json + +{ + "tenant": "public", + "namespace": "default", + "name": "pulsar-postgres-jdbc-sink", + "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink", + "inputSpecs": { + "pulsar-postgres-jdbc-sink-topic": { + "isRegexPattern": false + } + }, + "configs": { + "password": "password", + "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", + "userName": "postgres", + "tableName": "pulsar_postgres_jdbc_sink" + }, + "parallelism": 2, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true +} + +``` + +### Delete a JDBC sink + +You can use the [Connector Admin CLI](io-cli) +to delete a connector and perform other operations on it. + +This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. + +```bash + +$ bin/pulsar-admin sinks delete \ +--tenant public \ +--namespace default \ +--name pulsar-postgres-jdbc-sink + +``` + +:::tip + + +For more information about `pulsar-admin sinks delete options`, see [here](io-cli.md/#delete-1). + +::: + +The sink connector has been deleted successfully if the following message appears. + +```text + +"Deleted successfully" + +``` + +This example double-checks the status of the sink connector. + +```bash + +$ bin/pulsar-admin sinks get \ +--tenant public \ +--namespace default \ +--name pulsar-postgres-jdbc-sink + +``` + +The result shows that the sink connector does not exist. + +```text + +HTTP 404 Not Found + +Reason: Sink pulsar-postgres-jdbc-sink doesn't exist + +``` diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-use.md b/site2/website-next/versioned_docs/version-2.7.2/io-use.md new file mode 100644 index 0000000000000..1c98a98227d7a --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-use.md @@ -0,0 +1,1957 @@ +--- +id: io-use +title: How to use Pulsar connectors +sidebar_label: Use +original_id: io-use +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +This guide describes how to use Pulsar connectors. + +## Install a connector + +Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. + +:::note + + +When using a non-builtin connector, you need to specify the path of a archive file for the connector. + +::: + +To set up a builtin connector, follow +the instructions [here](getting-started-standalone.md#installing-builtin-connectors). + +After the setup, the builtin connector is automatically discovered by Pulsar brokers (or function-workers), so no additional installation steps are required. + +## Configure a connector + +You can configure the following information: + +* [Configure a default storage location for a connector](#configure-a-default-storage-location-for-a-connector) + +* [Configure a connector with a YAML file](#configure-a-connector-with-yaml-file) + +### Configure a default storage location for a connector + +To configure a default folder for builtin connectors, set the `connectorsDirectory` parameter in the `./conf/functions_worker.yml` configuration file. + +**Example** + +Set the `./connectors` folder as the default storage location for builtin connectors. + +``` +######################## +# Connectors +######################## + +connectorsDirectory: ./connectors + +``` + +### Configure a connector with a YAML file + +To configure a connector, you need to provide a YAML configuration file when creating a connector. + +The YAML configuration file tells Pulsar where to locate connectors and how to connect connectors with Pulsar topics. + +**Example 1** + +Below is a YAML configuration file of a Cassandra sink, which tells Pulsar: + +* Which Cassandra cluster to connect + +* What is the `keyspace` and `columnFamily` to be used in Cassandra for collecting data + +* How to map Pulsar messages into Cassandra table key and columns + +```shell + +tenant: public +namespace: default +name: cassandra-test-sink +... +# cassandra specific config +configs: + roots: "localhost:9042" + keyspace: "pulsar_test_keyspace" + columnFamily: "pulsar_test_table" + keyname: "key" + columnName: "col" + +``` + +**Example 2** + +Below is a YAML configuration file of a Kafka source. + +```shell + +configs: + bootstrapServers: "pulsar-kafka:9092" + groupId: "test-pulsar-io" + topic: "my-topic" + sessionTimeoutMs: "10000" + autoCommitEnabled: "false" + +``` + +**Example 3** + +Below is a YAML configuration file of a PostgreSQL JDBC sink. + +```shell + +configs: + userName: "postgres" + password: "password" + jdbcUrl: "jdbc:postgresql://localhost:5432/test_jdbc" + tableName: "test_jdbc" + +``` + +## Get available connectors + +Before starting using connectors, you can perform the following operations: + +* [Reload connectors](#reload) + +* [Get a list of available connectors](#get-available-connectors) + +### `reload` + +If you add or delete a nar file in a connector folder, reload the available builtin connector before using it. + +#### Source + +Use the `reload` subcommand. + +```shell + +$ pulsar-admin sources reload + +``` + +For more information, see [`here`](io-cli.md#reload). + +#### Sink + +Use the `reload` subcommand. + +```shell + +$ pulsar-admin sinks reload + +``` + +For more information, see [`here`](io-cli.md#reload-1). + +### `available` + +After reloading connectors (optional), you can get a list of available connectors. + +#### Source + +Use the `available-sources` subcommand. + +```shell + +$ pulsar-admin sources available-sources + +``` + +#### Sink + +Use the `available-sinks` subcommand. + +```shell + +$ pulsar-admin sinks available-sinks + +``` + +## Run a connector + +To run a connector, you can perform the following operations: + +* [Create a connector](#create) + +* [Start a connector](#start) + +* [Run a connector locally](#localrun) + +### `create` + +You can create a connector using **Admin CLI**, **REST API** or **JAVA admin API**.f + +#### Source + +Create a source connector. + + + + + +Use the `create` subcommand. + +``` +$ pulsar-admin sources create options + +``` + +For more information, see [here](io-cli.md#create). + + + + +Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/registerSource?version=@pulsar:version_number@} + + + + +* Create a source connector with a **local file**. + + ```java + + void createSource(SourceConfig sourceConfig, + String fileName) + throws PulsarAdminException + + ``` + + **Parameter** + + |Name|Description + |---|--- + `sourceConfig` | The source configuration object + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`createSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSource-SourceConfig-java.lang.String-). + +* Create a source connector using a **remote file** with a URL from which fun-pkg can be downloaded. + + ```java + + void createSourceWithUrl(SourceConfig sourceConfig, + String pkgUrl) + throws PulsarAdminException + + ``` + + Supported URLs are `http` and `file`. + + **Example** + + * HTTP: http://www.repo.com/fileName.jar + + * File: file:///dir/fileName.jar + + **Parameter** + + Parameter| Description + |---|--- + `sourceConfig` | The source configuration object + `pkgUrl` | URL from which pkg can be downloaded + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSourceWithUrl-SourceConfig-java.lang.String-). + + + + + +#### Sink + +Create a sink connector. + + + + + +Use the `create` subcommand. + +``` +$ pulsar-admin sinks create options + +``` + +For more information, see [here](io-cli.md#create-1). + + + + +Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sinkName|operation/registerSink?version=@pulsar:version_number@} + + + + +* Create a sink connector with a **local file**. + + + ```java + + void createSink(SinkConfig sinkConfig, + String fileName) + throws PulsarAdminException + + ``` + + **Parameter** + + |Name|Description + |---|--- + `sinkConfig` | The sink configuration object + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`createSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSink-SinkConfig-java.lang.String-). + +* Create a sink connector using a **remote file** with a URL from which fun-pkg can be downloaded. + + ```java + + void createSinkWithUrl(SinkConfig sinkConfig, + String pkgUrl) + throws PulsarAdminException + + ``` + + Supported URLs are `http` and `file`. + + **Example** + + * HTTP: http://www.repo.com/fileName.jar + + * File: file:///dir/fileName.jar + + **Parameter** + + Parameter| Description + |---|--- + `sinkConfig` | The sink configuration object + `pkgUrl` | URL from which pkg can be downloaded + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`createSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSinkWithUrl-SinkConfig-java.lang.String-). + + + + + +### `start` + +You can start a connector using **Admin CLI** or **REST API**. + +#### Source + +Start a source connector. + + + + + +Use the `start` subcommand. + +``` +$ pulsar-admin sources start options + +``` + +For more information, see [here](io-cli.md#start). + + + + +* Start **all** source connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/start|operation/startSource?version=@pulsar:version_number@} + +* Start a **specified** source connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSource?version=@pulsar:version_number@} + + + + + +#### Sink + +Start a sink connector. + + + + + +Use the `start` subcommand. + +``` +$ pulsar-admin sinks start options + +``` + +For more information, see [here](io-cli.md#start-1). + + + + +* Start **all** sink connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/start|operation/startSink?version=@pulsar:version_number@} + +* Start a **specified** sink connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSink?version=@pulsar:version_number@} + + + + + +### `localrun` + +You can run a connector locally rather than deploying it on a Pulsar cluster using **Admin CLI**. + +#### Source + +Run a source connector locally. + + + + + +Use the `localrun` subcommand. + +``` +$ pulsar-admin sources localrun options + +``` + +For more information, see [here](io-cli.md#localrun). + + + + + +#### Sink + +Run a sink connector locally. + + + + + +Use the `localrun` subcommand. + +``` +$ pulsar-admin sinks localrun options + +``` + +For more information, see [here](io-cli.md#localrun-1). + + + + + +## Monitor a connector + +To monitor a connector, you can perform the following operations: + +* [Get the information of a connector](#get) + +* [Get the list of all running connectors](#list) + +* [Get the current status of a connector](#status) + +### `get` + +You can get the information of a connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Get the information of a source connector. + + + + + +Use the `get` subcommand. + +``` +$ pulsar-admin sources get options + +``` + +For more information, see [here](io-cli.md#get). + + + + +Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/getSourceInfo?version=@pulsar:version_number@} + + + + +```java + +SourceConfig getSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + +``` + +**Example** + +This is a sourceConfig. + +```java + +{ + "tenant": "tenantName", + "namespace": "namespaceName", + "name": "sourceName", + "className": "className", + "topicName": "topicName", + "configs": {}, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "resources": { + "cpu": 1.0, + "ram": 1073741824, + "disk": 10737418240 + } +} + +``` + +This is a sourceConfig example. + +``` +{ + "tenant": "public", + "namespace": "default", + "name": "debezium-mysql-source", + "className": "org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource", + "topicName": "debezium-mysql-topic", + "configs": { + "database.user": "debezium", + "database.server.id": "184054", + "database.server.name": "dbserver1", + "database.port": "3306", + "database.hostname": "localhost", + "database.password": "dbz", + "database.history.pulsar.service.url": "pulsar://127.0.0.1:6650", + "value.converter": "org.apache.kafka.connect.json.JsonConverter", + "database.whitelist": "inventory", + "key.converter": "org.apache.kafka.connect.json.JsonConverter", + "database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory", + "pulsar.service.url": "pulsar://127.0.0.1:6650", + "database.history.pulsar.topic": "history-topic2" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "resources": { + "cpu": 1.0, + "ram": 1073741824, + "disk": 10737418240 + } +} + +``` + +**Exception** + +Exception name | Description +|---|--- +`PulsarAdminException.NotAuthorizedException` | You don't have the admin permission +`PulsarAdminException.NotFoundException` | Cluster doesn't exist +`PulsarAdminException` | Unexpected error + +For more information, see [`getSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSource-java.lang.String-java.lang.String-java.lang.String-). + + + + + +#### Sink + +Get the information of a sink connector. + + + + + +Use the `get` subcommand. + +``` +$ pulsar-admin sinks get options + +``` + +For more information, see [here](io-cli.md#get-1). + + + + +Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sinks/:tenant/:namespace/:sinkName|operation/getSinkInfo?version=@pulsar:version_number@} + + + + +```java + +SinkConfig getSink(String tenant, + String namespace, + String sink) + throws PulsarAdminException + +``` + +**Example** + +This is a sinkConfig. + +```json + +{ +"tenant": "tenantName", +"namespace": "namespaceName", +"name": "sinkName", +"className": "className", +"inputSpecs": { +"topicName": { + "isRegexPattern": false +} +}, +"configs": {}, +"parallelism": 1, +"processingGuarantees": "ATLEAST_ONCE", +"retainOrdering": false, +"autoAck": true +} + +``` + +This is a sinkConfig example. + +```json + +{ + "tenant": "public", + "namespace": "default", + "name": "pulsar-postgres-jdbc-sink", + "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink", + "inputSpecs": { + "pulsar-postgres-jdbc-sink-topic": { + "isRegexPattern": false + } + }, + "configs": { + "password": "password", + "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", + "userName": "postgres", + "tableName": "pulsar_postgres_jdbc_sink" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true +} + +``` + +**Parameter description** + +Name| Description +|---|--- +`tenant` | Tenant name +`namespace` | Namespace name +`sink` | Sink name + +For more information, see [`getSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSink-java.lang.String-java.lang.String-java.lang.String-). + + + + + +### `list` + +You can get the list of all running connectors using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Get the list of all running source connectors. + + + + + +Use the `list` subcommand. + +``` +$ pulsar-admin sources list options + +``` + +For more information, see [here](io-cli.md#list). + + + + +Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sources/:tenant/:namespace/|operation/listSources?version=@pulsar:version_number@} + + + + +```java + +List listSources(String tenant, + String namespace) + throws PulsarAdminException + +``` + +**Response example** + +```java ["f1", "f2", "f3"] + +``` + +**Exception** + +Exception name | Description +|---|--- +`PulsarAdminException.NotAuthorizedException` | You don't have the admin permission +`PulsarAdminException` | Unexpected error + +For more information, see [`listSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#listSources-java.lang.String-java.lang.String-). + + + + + +#### Sink + +Get the list of all running sink connectors. + + + + + +Use the `list` subcommand. + +``` +$ pulsar-admin sinks list options + +``` + +For more information, see [here](io-cli.md#list-1). + + + + +Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sinks/:tenant/:namespace/|operation/listSinks?version=@pulsar:version_number@} + + + + +```java + +List listSinks(String tenant, + String namespace) + throws PulsarAdminException + +``` + +**Response example** + +```java ["f1", "f2", "f3"] + +``` + +**Exception** + +Exception name | Description +|---|--- +`PulsarAdminException.NotAuthorizedException` | You don't have the admin permission +`PulsarAdminException` | Unexpected error + +For more information, see [`listSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#listSinks-java.lang.String-java.lang.String-). + + + + + +### `status` + +You can get the current status of a connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Get the current status of a source connector. + + + + + +Use the `status` subcommand. + +``` +$ pulsar-admin sources status options + +``` + +For more information, see [here](io-cli.md#status). + + + + +* Get the current status of **all** source connectors. + + Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sources/:tenant/:namespace/:sourceName/status|operation/getSourceStatus?version=@pulsar:version_number@} + +* Gets the current status of a **specified** source connector. + + Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/status|operation/getSourceStatus?version=@pulsar:version_number@} + + + + +* Get the current status of **all** source connectors. + + ```java + + SourceStatus getSourceStatus(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` + + **Parameter** + + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + + **Exception** + + Name | Description + |---|--- + `PulsarAdminException` | Unexpected error + + For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSource-java.lang.String-java.lang.String-java.lang.String-). + +* Gets the current status of a **specified** source connector. + + ```java + + SourceStatus.SourceInstanceStatus.SourceInstanceStatusData getSourceStatus(String tenant, + String namespace, + String source, + int id) + throws PulsarAdminException + + ``` + + **Parameter** + + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + `id` | Source instanceID + + **Exception** + + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error + + For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSourceStatus-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +#### Sink + +Get the current status of a Pulsar sink connector. + + + + + +Use the `status` subcommand. + +``` +$ pulsar-admin sinks status options + +``` + +For more information, see [here](io-cli.md#status-1). + + + + +* Get the current status of **all** sink connectors. + + Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sinks/:tenant/:namespace/:sinkName/status|operation/getSinkStatus?version=@pulsar:version_number@} + +* Gets the current status of a **specified** sink connector. + + Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v3/sinks/:tenant/:namespace/:sourceName/:instanceId/status|operation/getSinkInstanceStatus?version=@pulsar:version_number@} + + + + +* Get the current status of **all** sink connectors. + + ```java + + SinkStatus getSinkStatus(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` + + **Parameter** + + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + + **Exception** + + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error + + For more information, see [`getSinkStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-). + +* Gets the current status of a **specified** source connector. + + ```java + + SinkStatus.SinkInstanceStatus.SinkInstanceStatusData getSinkStatus(String tenant, + String namespace, + String sink, + int id) + throws PulsarAdminException + + ``` + + **Parameter** + + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + `id` | Sink instanceID + + **Exception** + + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error + + For more information, see [`getSinkStatusWithInstanceID`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +## Update a connector + +### `update` + +You can update a running connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Update a running Pulsar source connector. + + + + + +Use the `update` subcommand. + +``` +$ pulsar-admin sources update options + +``` + +For more information, see [here](io-cli.md#update). + + + + +Send a `PUT` request to this endpoint: {@inject: endpoint|PUT|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/updateSource?version=@pulsar:version_number@} + + + + +* Update a running source connector with a **local file**. + + ```java + + void updateSource(SourceConfig sourceConfig, + String fileName) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + |`sourceConfig` | The source configuration object + + **Exception** + + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error + + For more information, see [`updateSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#updateSource-SourceConfig-java.lang.String-). + +* Update a source connector using a **remote file** with a URL from which fun-pkg can be downloaded. + + ```java + + void updateSourceWithUrl(SourceConfig sourceConfig, + String pkgUrl) + throws PulsarAdminException + + ``` + + Supported URLs are `http` and `file`. + + **Example** + + * HTTP: http://www.repo.com/fileName.jar + + * File: file:///dir/fileName.jar + + **Parameter** + + | Name | Description + |---|--- + | `sourceConfig` | The source configuration object + | `pkgUrl` | URL from which pkg can be downloaded + + **Exception** + + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error + +For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#updateSourceWithUrl-SourceConfig-java.lang.String-). + + + + + +#### Sink + +Update a running Pulsar sink connector. + + + + + +Use the `update` subcommand. + +``` +$ pulsar-admin sinks update options + +``` + +For more information, see [here](io-cli.md#update-1). + + + + +Send a `PUT` request to this endpoint: {@inject: endpoint|PUT|/admin/v3/sinks/:tenant/:namespace/:sinkName|operation/updateSink?version=@pulsar:version_number@} + + + + +* Update a running sink connector with a **local file**. + + ```java + + void updateSink(SinkConfig sinkConfig, + String fileName) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + |`sinkConfig` | The sink configuration object + + **Exception** + + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error + + For more information, see [`updateSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#updateSink-SinkConfig-java.lang.String-). + +* Update a sink connector using a **remote file** with a URL from which fun-pkg can be downloaded. + + ```java + + void updateSinkWithUrl(SinkConfig sinkConfig, + String pkgUrl) + throws PulsarAdminException + + ``` + + Supported URLs are `http` and `file`. + + **Example** + + * HTTP: http://www.repo.com/fileName.jar + + * File: file:///dir/fileName.jar + + **Parameter** + + | Name | Description + |---|--- + | `sinkConfig` | The sink configuration object + | `pkgUrl` | URL from which pkg can be downloaded + + **Exception** + + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + |`PulsarAdminException.NotFoundException` | Cluster doesn't exist + |`PulsarAdminException` | Unexpected error + +For more information, see [`updateSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#updateSinkWithUrl-SinkConfig-java.lang.String-). + + + + + +## Stop a connector + +### `stop` + +You can stop a connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Stop a source connector. + + + + + +Use the `stop` subcommand. + +``` +$ pulsar-admin sources stop options + +``` + +For more information, see [here](io-cli.md#stop). + + + + +* Stop **all** source connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/stopSource?version=@pulsar:version_number@} + +* Stop a **specified** source connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId|operation/stopSource?version=@pulsar:version_number@} + + + + +* Stop **all** source connectors. + + ```java + + void stopSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-). + +* Stop a **specified** source connector. + + ```java + + void stopSource(String tenant, + String namespace, + String source, + int instanceId) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +#### Sink + +Stop a sink connector. + + + + + +Use the `stop` subcommand. + +``` +$ pulsar-admin sinks stop options + +``` + +For more information, see [here](io-cli.md#stop-1). + + + + +* Stop **all** sink connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sinkName/stop|operation/stopSink?version=@pulsar:version_number@} + +* Stop a **specified** sink connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkeName/:instanceId/stop|operation/stopSink?version=@pulsar:version_number@} + + + + +* Stop **all** sink connectors. + + ```java + + void stopSink(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-). + +* Stop a **specified** sink connector. + + ```java + + void stopSink(String tenant, + String namespace, + String sink, + int instanceId) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +## Restart a connector + +### `restart` + +You can restart a connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Restart a source connector. + + + + + +Use the `restart` subcommand. + +``` +$ pulsar-admin sources restart options + +``` + +For more information, see [here](io-cli.md#restart). + + + + +* Restart **all** source connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/restart|operation/restartSource?version=@pulsar:version_number@} + +* Restart a **specified** source connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/restart|operation/restartSource?version=@pulsar:version_number@} + + + + +* Restart **all** source connectors. + + ```java + + void restartSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-). + +* Restart a **specified** source connector. + + ```java + + void restartSource(String tenant, + String namespace, + String source, + int instanceId) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +#### Sink + +Restart a sink connector. + + + + + +Use the `restart` subcommand. + +``` +$ pulsar-admin sinks restart options + +``` + +For more information, see [here](io-cli.md#restart-1). + + + + +* Restart **all** sink connectors. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/restart|operation/restartSource?version=@pulsar:version_number@} + +* Restart a **specified** sink connector. + + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/:instanceId/restart|operation/restartSource?version=@pulsar:version_number@} + + + + +* Restart all Pulsar sink connectors. + + ```java + + void restartSink(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Sink name + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-). + +* Restart a **specified** sink connector. + + ```java + + void restartSink(String tenant, + String namespace, + String sink, + int instanceId) + throws PulsarAdminException + + ``` + + **Parameter** + + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Sink instanceID + + **Exception** + + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error + + For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-int-). + + + + + +## Delete a connector + +### `delete` + +You can delete a connector using **Admin CLI**, **REST API** or **JAVA admin API**. + +#### Source + +Delete a source connector. + + + + + +Use the `delete` subcommand. + +``` +$ pulsar-admin sources delete options + +``` + +For more information, see [here](io-cli.md#delete). + + + + +Delete al Pulsar source connector. + +Send a `DELETE` request to this endpoint: {@inject: endpoint|DELETE|/admin/v3/sources/:tenant/:namespace/:sourceName|operation/deregisterSource?version=@pulsar:version_number@} + + + + +Delete a source connector. + +```java + +void deleteSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + +``` + +**Parameter** + +| Name | Description +|---|--- +`tenant` | Tenant name +`namespace` | Namespace name +`source` | Source name + +**Exception** + +|Name|Description| +|---|--- +|`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission +| `PulsarAdminException.NotFoundException` | Cluster doesn't exist +| `PulsarAdminException.PreconditionFailedException` | Cluster is not empty +| `PulsarAdminException` | Unexpected error + +For more information, see [`deleteSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#deleteSource-java.lang.String-java.lang.String-java.lang.String-). + + + + + +#### Sink + +Delete a sink connector. + + + + + +Use the `delete` subcommand. + +``` +$ pulsar-admin sinks delete options + +``` + +For more information, see [here](io-cli.md#delete-1). + + + + +Delete a sink connector. + +Send a `DELETE` request to this endpoint: {@inject: endpoint|DELETE|/admin/v3/sinks/:tenant/:namespace/:sinkName|operation/deregisterSink?version=@pulsar:version_number@} + + + + +Delete a Pulsar sink connector. + +```java + +void deleteSink(String tenant, + String namespace, + String source) + throws PulsarAdminException + +``` + +**Parameter** + +| Name | Description +|---|--- +`tenant` | Tenant name +`namespace` | Namespace name +`sink` | Sink name + +**Exception** + +|Name|Description| +|---|--- +|`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission +| `PulsarAdminException.NotFoundException` | Cluster doesn't exist +| `PulsarAdminException.PreconditionFailedException` | Cluster is not empty +| `PulsarAdminException` | Unexpected error + +For more information, see [`deleteSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#deleteSink-java.lang.String-java.lang.String-java.lang.String-). + + + + diff --git a/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json b/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json index e8c9dbbc07207..5e86333ebca86 100644 --- a/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json +++ b/site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json @@ -85,6 +85,44 @@ "id": "version-2.7.2/schema-manage" } ] + }, + { + "type": "category", + "label": "Pulsar IO", + "items": [ + { + "type": "doc", + "id": "version-2.7.2/io-overview" + }, + { + "type": "doc", + "id": "version-2.7.2/io-quickstart" + }, + { + "type": "doc", + "id": "version-2.7.2/io-use" + }, + { + "type": "doc", + "id": "version-2.7.2/io-debug" + }, + { + "type": "doc", + "id": "version-2.7.2/io-connectors" + }, + { + "type": "doc", + "id": "version-2.7.2/io-cdc" + }, + { + "type": "doc", + "id": "version-2.7.2/io-develop" + }, + { + "type": "doc", + "id": "version-2.7.2/io-cli" + } + ] } ] } \ No newline at end of file From ef7469d4be44848e77c545791af5b7ff06a8985a Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Wed, 20 Oct 2021 23:40:22 +0800 Subject: [PATCH 3/8] Update the nesting of forms in the document --- .../schema-evolution-compatibility.md | 24 +++++++++--------- .../version-2.7.2/schema-get-started.md | 2 +- .../version-2.7.2/schema-manage.md | 25 ++++++++----------- .../version-2.7.2/schema-understand.md | 8 +++--- 4 files changed, 28 insertions(+), 31 deletions(-) diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md b/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md index 60a436cb4b720..a2bb99922254f 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md @@ -1,7 +1,7 @@ --- id: schema-evolution-compatibility title: Schema evolution and compatibility -sidebar_label: Schema evolution and compatibility +sidebar_label: "Schema evolution and compatibility" original_id: schema-evolution-compatibility --- @@ -59,19 +59,19 @@ Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is t | --- | --- | --- | --- | --- | | `ALWAYS_COMPATIBLE` | Disable schema compatibility check. | All changes are allowed | All previous versions | Any order | | `ALWAYS_INCOMPATIBLE` | Disable schema evolution. | All changes are disabled | None | None | -| `BACKWARD` | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. | * Add optional fields * Delete fields | Latest version | Consumers | -| `BACKWARD_TRANSITIVE` | Consumers using the schema V3 can process data written by producers using the schema V3, V2 or V1. | * Add optional fields * Delete fields | All previous versions | Consumers | -| `FORWARD` | Consumers using the schema V3 or V2 can process data written by producers using the schema V3. | * Add fields * Delete optional fields | Latest version | Producers | -| `FORWARD_TRANSITIVE` | Consumers using the schema V3, V2 or V1 can process data written by producers using the schema V3. | * Add fields * Delete optional fields | All previous versions | Producers | -| `FULL` | Backward and forward compatible between the schema V3 and V2. | * Modify optional fields | Latest version | Any order | -| `FULL_TRANSITIVE` | Backward and forward compatible among the schema V3, V2, and V1. | * Modify optional fields | All previous versions | Any order | +| `BACKWARD` | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. |
  • Add optional fields
  • Delete fields
    • | Latest version | Consumers | +| `BACKWARD_TRANSITIVE` | Consumers using the schema V3 can process data written by producers using the schema V3, V2 or V1. |
  • Add optional fields
  • Delete fields
    • | All previous versions | Consumers | +| `FORWARD` | Consumers using the schema V3 or V2 can process data written by producers using the schema V3. |
  • Add fields
  • Delete optional fields
    • | Latest version | Producers | +| `FORWARD_TRANSITIVE` | Consumers using the schema V3, V2 or V1 can process data written by producers using the schema V3. |
  • Add fields
  • Delete optional fields
    • | All previous versions | Producers | +| `FULL` | Backward and forward compatible between the schema V3 and V2. |
  • Modify optional fields
    • | Latest version | Any order | +| `FULL_TRANSITIVE` | Backward and forward compatible among the schema V3, V2, and V1. |
  • Modify optional fields
    • | All previous versions | Any order | ### ALWAYS_COMPATIBLE and ALWAYS_INCOMPATIBLE | Compatibility check strategy | Definition | Note | | --- | --- | --- | | `ALWAYS_COMPATIBLE` | Disable schema compatibility check. | None | -| `ALWAYS_INCOMPATIBLE` | Disable schema evolution, that is, any schema change is rejected. | * For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`. * For Avro and JSON, the default schema compatibility check strategy is `FULL`. | +| `ALWAYS_INCOMPATIBLE` | Disable schema evolution, that is, any schema change is rejected. |
  • For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`.
  • For Avro and JSON, the default schema compatibility check strategy is `FULL`.
    • | #### Example @@ -151,7 +151,7 @@ Suppose that you have a topic containing three schemas (V1, V2, and V3), V1 is t | Compatibility check strategy | Definition | Description | Note | | --- | --- | --- | --- | -| `FULL` | Schemas are both backward and forward compatible, which means: Consumers using the last schema can process data written by producers using the new schema. AND Consumers using the new schema can process data written by producers using the last schema. | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. AND Consumers using the schema V3 or V2 can process data written by producers using the schema V3. | * For Avro and JSON, the default schema compatibility check strategy is `FULL`. * For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`. | +| `FULL` | Schemas are both backward and forward compatible, which means: Consumers using the last schema can process data written by producers using the new schema. AND Consumers using the new schema can process data written by producers using the last schema. | Consumers using the schema V3 can process data written by producers using the schema V3 or V2. AND Consumers using the schema V3 or V2 can process data written by producers using the schema V3. |
  • For Avro and JSON, the default schema compatibility check strategy is `FULL`.
  • For all schema types except Avro and JSON, the default schema compatibility check strategy is `ALWAYS_INCOMPATIBLE`.
    • | | `FULL_TRANSITIVE` | The new schema is backward and forward compatible with all previously registered schemas. | Consumers using the schema V3 can process data written by producers using the schema V3, V2 or V1. AND Consumers using the schema V3, V2 or V1 can process data written by producers using the schema V3. | None | #### Example @@ -196,9 +196,9 @@ For example, the producers using schemas to write data to Pulsar and the consume | --- | --- | --- | | `ALWAYS_COMPATIBLE` | Any order | The compatibility check is disabled. Consequently, you can upgrade the producers and consumers in **any order**. | | `ALWAYS_INCOMPATIBLE` | None | The schema evolution is disabled. | -| * `BACKWARD` * `BACKWARD_TRANSITIVE` | Consumers | There is no guarantee that consumers using the old schema can read data produced using the new schema. Consequently, **upgrade all consumers first**, and then start producing new data. | -| * `FORWARD` * `FORWARD_TRANSITIVE` | Producers | There is no guarantee that consumers using the new schema can read data produced using the old schema. Consequently, **upgrade all producers first** to use the new schema and ensure that the data already produced using the old schemas are not available to consumers, and then upgrade the consumers. | -| * `FULL` * `FULL_TRANSITIVE` | Any order | There is no guarantee that consumers using the old schema can read data produced using the new schema and consumers using the new schema can read data produced using the old schema. Consequently, you can upgrade the producers and consumers in **any order**. | +|
  • `BACKWARD`
  • `BACKWARD_TRANSITIVE`
    • | Consumers | There is no guarantee that consumers using the old schema can read data produced using the new schema. Consequently, **upgrade all consumers first**, and then start producing new data. | +|
  • `FORWARD`
  • `FORWARD_TRANSITIVE`
    • | Producers | There is no guarantee that consumers using the new schema can read data produced using the old schema. Consequently, **upgrade all producers first**
  • to use the new schema and ensure that the data already produced using the old schemas are not available to consumers, and then upgrade the consumers.
    • | +|
  • `FULL`
  • `FULL_TRANSITIVE`
    • | Any order | There is no guarantee that consumers using the old schema can read data produced using the new schema and consumers using the new schema can read data produced using the old schema. Consequently, you can upgrade the producers and consumers in **any order**. | diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md index 496fe9077e491..c8193ab572c69 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md @@ -1,7 +1,7 @@ --- id: schema-get-started title: Get started -sidebar_label: Get started +sidebar_label: "Get started" original_id: schema-get-started --- diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md b/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md index aa42485736939..e95369f0da08a 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-manage.md @@ -1,7 +1,7 @@ --- id: schema-manage title: Manage schema -sidebar_label: Manage schema +sidebar_label: "Manage schema" original_id: schema-manage --- @@ -158,9 +158,9 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI** | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | -| **REST API** | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | -| **Java Admin API** | Pulsar provides Java admin library. | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | +| **Java Admin API**
  • | Pulsar provides Java admin library. | ### Upload a schema @@ -210,7 +210,7 @@ The `schema-definition-file` includes the following fields: | Field | Description | | --- | --- | | `type` | The schema type. | -| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this field should be blank.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition.
    • | | `properties` | The additional properties associated with the schema. | Here are examples of the `schema-definition-file` for a JSON schema. @@ -263,7 +263,7 @@ The post payload includes the following fields: | Field | Description | | --- | --- | | `type` | The schema type. | -| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this field should be blank.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition.
    • | | `properties` | The additional properties associated with the schema. | @@ -280,7 +280,7 @@ The `PostSchemaPayload` includes the following fields: | Field | Description | | --- | --- | | `type` | The schema type. | -| `schema` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `schema` | The schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this field should be blank.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition.
    • | | `properties` | The additional properties associated with the schema. | Here is an example of `PostSchemaPayload`: @@ -368,7 +368,7 @@ The response includes the following fields: | `version` | The schema version, which is a long number. | | `type` | The schema type. | | `timestamp` | The timestamp of creating this version of schema. | -| `data` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `data` | The schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this field should be blank.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition.
    • | | `properties` | The additional properties associated with the schema. | @@ -386,7 +386,7 @@ The `SchemaInfo` includes the following fields: | --- | --- | | `name` | The schema name. | | `type` | The schema type. | -| `schema` | A byte array of the schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this byte array should be empty. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition converted to a byte array. | +| `schema` | A byte array of the schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this byte array should be empty.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition converted to a byte array.
    • | | `properties` | The additional properties associated with the schema. | Here is an example of `SchemaInfo`: @@ -460,7 +460,7 @@ The response includes the following fields: | `version` | The schema version, which is a long number. | | `type` | The schema type. | | `timestamp` | The timestamp of creating this version of schema. | -| `data` | The schema definition data, which is encoded in UTF 8 charset. * If the schema is a **primitive** schema, this field should be blank. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition. | +| `data` | The schema definition data, which is encoded in UTF 8 charset.
  • If the schema is a
    • **primitive**
  • schema, this field should be blank.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition.
    • | | `properties` | The additional properties associated with the schema. | @@ -478,7 +478,7 @@ The `SchemaInfo` includes the following fields: | --- | --- | | `name` | The schema name. | | `type` | The schema type. | -| `schema` | A byte array of the schema definition data, which is encoded in UTF 8. * If the schema is a **primitive** schema, this byte array should be empty. * If the schema is a **struct** schema, this field should be a JSON string of the Avro schema definition converted to a byte array. | +| `schema` | A byte array of the schema definition data, which is encoded in UTF 8.
  • If the schema is a
    • **primitive**
  • schema, this byte array should be empty.
  • If the schema is a
    • **struct**
  • schema, this field should be a JSON string of the Avro schema definition converted to a byte array.
    • | | `properties` | The additional properties associated with the schema. | Here is an example of `SchemaInfo`: @@ -528,7 +528,6 @@ To delete a schema for a topic, you can use one of the following methods. :::note - In any case, the **delete** action deletes **all versions** of a schema registered for a topic. ::: @@ -648,7 +647,6 @@ public interface SchemaStorage { :::tip - For a complete example of **schema storage** implementation, see [BookKeeperSchemaStorage](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java) class. ::: @@ -668,7 +666,6 @@ public interface SchemaStorageFactory { :::tip - For a complete example of **schema storage factory** implementation, see [BookKeeperSchemaStorageFactory](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorageFactory.java) class. ::: diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md b/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md index 7de41e6c3838f..87822c55810b4 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-understand.md @@ -1,7 +1,7 @@ --- id: schema-understand title: Understand schema -sidebar_label: Understand schema +sidebar_label: "Understand schema" original_id: schema-understand --- @@ -22,7 +22,7 @@ A `SchemaInfo` consists of the following fields: | Field | Description | | --- | --- | | `name` | Schema name (a string). | -| `type` | Schema type, which determines how to interpret the schema data. * Predefined schema: see [here](schema-understand.md#schema-type). * Customized schema: it is left as an empty string. | +| `type` | Schema type, which determines how to interpret the schema data.
  • Predefined schema: see [here](schema-understand.md#schema-type).
  • Customized schema: it is left as an empty string.
    • | | `schema`（`payload`) | Schema data, which is a sequence of 8-bit unsigned bytes and schema-type specific. | | `properties` | It is a user defined properties as a string/string map. Applications can use this bag for carrying any application specific logics. Possible properties might be the Git hash associated with the schema, an environment string like `dev` or `prod`. | @@ -403,8 +403,8 @@ The table below lists the possible scenarios when this connection attempt occurs | Scenario | What happens | | --- | --- | -| * No schema exists for the topic. | (1) The producer is created using the given schema. (2) Since no existing schema is compatible with the `SensorReading` schema, the schema is transmitted to the broker and stored. (3) Any consumer created using the same schema or topic can consume messages from the `sensor-data` topic. | -| * A schema already exists. * The producer connects using the same schema that is already stored. | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible. (3) The broker attempts to store the schema in [BookKeeper](concepts-architecture-overview.md#persistent-storage) but then determines that it's already stored, so it is used to tag produced messages. | * A schema already exists. * The producer connects using a new schema that is compatible. | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible and stores the new schema as the current version (with a new version number). | +|
  • No schema exists for the topic.
    • | (1) The producer is created using the given schema. (2) Since no existing schema is compatible with the `SensorReading` schema, the schema is transmitted to the broker and stored. (3) Any consumer created using the same schema or topic can consume messages from the `sensor-data` topic. | +|
  • A schema already exists.
  • The producer connects using the same schema that is already stored.
    • | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible. (3) The broker attempts to store the schema in [BookKeeper](concepts-architecture-overview.md#persistent-storage) but then determines that it's already stored, so it is used to tag produced messages. |
  • A schema already exists.
  • The producer connects using a new schema that is compatible.
    • | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible and stores the new schema as the current version (with a new version number). | ## How does schema work From d23dde2e8b0523cb6cb1f1f34a57767a5786491f Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Tue, 26 Oct 2021 15:01:45 +0800 Subject: [PATCH 4/8] update pulsar io --- .../version-2.7.2/io-aerospike-sink.md | 30 + .../version-2.7.2/io-canal-source.md | 239 +++++ .../version-2.7.2/io-cassandra-sink.md | 61 ++ .../version-2.7.2/io-cdc-debezium.md | 554 ++++++++++++ .../versioned_docs/version-2.7.2/io-cdc.md | 2 +- .../versioned_docs/version-2.7.2/io-cli.md | 2 +- .../version-2.7.2/io-connectors.md | 2 +- .../version-2.7.2/io-debezium-source.md | 589 +++++++++++++ .../versioned_docs/version-2.7.2/io-debug.md | 315 +++---- .../version-2.7.2/io-develop.md | 169 ++-- .../version-2.7.2/io-dynamodb-source.md | 84 ++ .../version-2.7.2/io-elasticsearch-sink.md | 177 ++++ .../version-2.7.2/io-file-source.md | 163 ++++ .../version-2.7.2/io-flume-sink.md | 60 ++ .../version-2.7.2/io-flume-source.md | 60 ++ .../version-2.7.2/io-hbase-sink.md | 70 ++ .../version-2.7.2/io-hdfs2-sink.md | 68 ++ .../version-2.7.2/io-hdfs3-sink.md | 63 ++ .../version-2.7.2/io-influxdb-sink.md | 123 +++ .../version-2.7.2/io-jdbc-sink.md | 161 ++++ .../version-2.7.2/io-kafka-sink.md | 76 ++ .../version-2.7.2/io-kafka-source.md | 201 +++++ .../version-2.7.2/io-kinesis-sink.md | 84 ++ .../version-2.7.2/io-kinesis-source.md | 85 ++ .../version-2.7.2/io-mongo-sink.md | 61 ++ .../version-2.7.2/io-netty-source.md | 245 ++++++ .../version-2.7.2/io-nsq-source.md | 25 + .../version-2.7.2/io-overview.md | 2 +- .../version-2.7.2/io-quickstart.md | 789 ++++++++--------- .../version-2.7.2/io-rabbitmq-sink.md | 89 ++ .../version-2.7.2/io-rabbitmq-source.md | 89 ++ .../version-2.7.2/io-redis-sink.md | 78 ++ .../version-2.7.2/io-solr-sink.md | 69 ++ .../version-2.7.2/io-twitter-source.md | 32 + .../version-2.7.2/io-twitter.md | 11 + .../versioned_docs/version-2.7.2/io-use.md | 828 +++++++++--------- 36 files changed, 4711 insertions(+), 1045 deletions(-) create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-aerospike-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-canal-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-cassandra-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-cdc-debezium.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-debezium-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-dynamodb-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-elasticsearch-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-file-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-flume-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-flume-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-hbase-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-hdfs2-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-hdfs3-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-influxdb-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-jdbc-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-kafka-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-kafka-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-kinesis-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-kinesis-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-mongo-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-netty-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-nsq-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-redis-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-solr-sink.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-twitter-source.md create mode 100644 site2/website-next/versioned_docs/version-2.7.2/io-twitter.md diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-aerospike-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-aerospike-sink.md new file mode 100644 index 0000000000000..7ff980521a489 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-aerospike-sink.md @@ -0,0 +1,30 @@ +--- +id: io-aerospike-sink +title: Aerospike sink connector +sidebar_label: "Aerospike sink connector" +original_id: io-aerospike-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Aerospike sink connector pulls messages from Pulsar topics to Aerospike clusters. + +## Configuration + +The configuration of the Aerospike sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `seedHosts` |String| true | No default value| The comma-separated list of one or more Aerospike cluster hosts.

    Each host can be specified as a valid IP address or hostname followed by an optional port number. | +| `keyspace` | String| true |No default value |The Aerospike namespace. | +| `columnName` | String | true| No default value|The Aerospike column name. | +|`userName`|String|false|NULL|The Aerospike username.| +|`password`|String|false|NULL|The Aerospike password.| +| `keySet` | String|false |NULL | The Aerospike set name. | +| `maxConcurrentRequests` |int| false | 100 | The maximum number of concurrent Aerospike transactions that a sink can open. | +| `timeoutMs` | int|false | 100 | This property controls `socketTimeout` and `totalTimeout` for Aerospike transactions. | +| `retries` | int|false | 1 |The maximum number of retries before aborting a write transaction to Aerospike. | diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-canal-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-canal-source.md new file mode 100644 index 0000000000000..853b387164d65 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-canal-source.md @@ -0,0 +1,239 @@ +--- +id: io-canal-source +title: Canal source connector +sidebar_label: "Canal source connector" +original_id: io-canal-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Canal source connector pulls messages from MySQL to Pulsar topics. + +## Configuration + +The configuration of Canal source connector has the following properties. + +### Property + +| Name | Required | Default | Description | +|------|----------|---------|-------------| +| `username` | true | None | Canal server account (not MySQL).| +| `password` | true | None | Canal server password (not MySQL). | +|`destination`|true|None|Source destination that Canal source connector connects to. +| `singleHostname` | false | None | Canal server address.| +| `singlePort` | false | None | Canal server port.| +| `cluster` | true | false | Whether to enable cluster mode based on Canal server configuration or not.

  • true: **cluster** mode.
    If set to true, it talks to `zkServers` to figure out the actual database host.

  • false: **standalone** mode.
    If set to false, it connects to the database specified by `singleHostname` and `singlePort`.
    • | +| `zkServers` | true | None | Address and port of the Zookeeper that Canal source connector talks to figure out the actual database host.| +| `batchSize` | false | 1000 | Batch size to fetch from Canal. | + +### Example + +Before using the Canal connector, you can create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "zkServers": "127.0.0.1:2181", + "batchSize": "5120", + "destination": "example", + "username": "", + "password": "", + "cluster": false, + "singleHostname": "127.0.0.1", + "singlePort": "11111", + } + + ``` + +* YAML + + You can create a YAML file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/resources/canal-mysql-source-config.yaml) below to your YAML file. + + ```yaml + + configs: + zkServers: "127.0.0.1:2181" + batchSize: 5120 + destination: "example" + username: "" + password: "" + cluster: false + singleHostname: "127.0.0.1" + singlePort: 11111 + + ``` + +## Usage + +Here is an example of storing MySQL data using the configuration file as above. + +1. Start a MySQL server. + + ```bash + + $ docker pull mysql:5.7 + $ docker run -d -it --rm --name pulsar-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=canal -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7 + + ``` + +2. Create a configuration file `mysqld.cnf`. + + ```bash + + [mysqld] + pid-file = /var/run/mysqld/mysqld.pid + socket = /var/run/mysqld/mysqld.sock + datadir = /var/lib/mysql + #log-error = /var/log/mysql/error.log + # By default we only accept connections from localhost + #bind-address = 127.0.0.1 + # Disabling symbolic-links is recommended to prevent assorted security risks + symbolic-links=0 + log-bin=mysql-bin + binlog-format=ROW + server_id=1 + + ``` + +3. Copy the configuration file `mysqld.cnf` to MySQL server. + + ```bash + + $ docker cp mysqld.cnf pulsar-mysql:/etc/mysql/mysql.conf.d/ + + ``` + +4. Restart the MySQL server. + + ```bash + + $ docker restart pulsar-mysql + + ``` + +5. Create a test database in MySQL server. + + ```bash + + $ docker exec -it pulsar-mysql /bin/bash + $ mysql -h 127.0.0.1 -uroot -pcanal -e 'create database test;' + + ``` + +6. Start a Canal server and connect to MySQL server. + + ``` + + $ docker pull canal/canal-server:v1.1.2 + $ docker run -d -it --link pulsar-mysql -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=pulsar-mysql:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=canal -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=pulsar-canal-server -p 8000:8000 -p 2222:2222 -p 11111:11111 -p 11112:11112 -m 4096m canal/canal-server:v1.1.2 + + ``` + +7. Start Pulsar standalone. + + ```bash + + $ docker pull apachepulsar/pulsar:2.3.0 + $ docker run -d -it --link pulsar-canal-server -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:2.3.0 bin/pulsar standalone + + ``` + +8. Modify the configuration file `canal-mysql-source-config.yaml`. + + ```yaml + + configs: + zkServers: "" + batchSize: "5120" + destination: "test" + username: "" + password: "" + cluster: false + singleHostname: "pulsar-canal-server" + singlePort: "11111" + + ``` + +9. Create a consumer file `pulsar-client.py`. + + ```python + + import pulsar + + client = pulsar.Client('pulsar://localhost:6650') + consumer = client.subscribe('my-topic', + subscription_name='my-sub') + + while True: + msg = consumer.receive() + print("Received message: '%s'" % msg.data()) + consumer.acknowledge(msg) + + client.close() + + ``` + +10. Copy the configuration file `canal-mysql-source-config.yaml` and the consumer file `pulsar-client.py` to Pulsar server. + + ```bash + + $ docker cp canal-mysql-source-config.yaml pulsar-standalone:/pulsar/conf/ + $ docker cp pulsar-client.py pulsar-standalone:/pulsar/ + + ``` + +11. Download a Canal connector and start it. + + ```bash + + $ docker exec -it pulsar-standalone /bin/bash + $ wget https://archive.apache.org/dist/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar -P connectors + $ ./bin/pulsar-admin source localrun \ + --archive ./connectors/pulsar-io-canal-2.3.0.nar \ + --classname org.apache.pulsar.io.canal.CanalStringSource \ + --tenant public \ + --namespace default \ + --name canal \ + --destination-topic-name my-topic \ + --source-config-file /pulsar/conf/canal-mysql-source-config.yaml \ + --parallelism 1 + + ``` + +12. Consume data from MySQL. + + ```bash + + $ docker exec -it pulsar-standalone /bin/bash + $ python pulsar-client.py + + ``` + +13. Open another window to log in MySQL server. + + ```bash + + $ docker exec -it pulsar-mysql /bin/bash + $ mysql -h 127.0.0.1 -uroot -pcanal + + ``` + +14. Create a table, and insert, delete, and update data in MySQL server. + + ```bash + + mysql> use test; + mysql> show tables; + mysql> CREATE TABLE IF NOT EXISTS `test_table`(`test_id` INT UNSIGNED AUTO_INCREMENT,`test_title` VARCHAR(100) NOT NULL, + `test_author` VARCHAR(40) NOT NULL, + `test_date` DATE,PRIMARY KEY ( `test_id` ))ENGINE=InnoDB DEFAULT CHARSET=utf8; + mysql> INSERT INTO test_table (test_title, test_author, test_date) VALUES("a", "b", NOW()); + mysql> UPDATE test_table SET test_title='c' WHERE test_title='a'; + mysql> DELETE FROM test_table WHERE test_title='c'; + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cassandra-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-cassandra-sink.md new file mode 100644 index 0000000000000..c79917ca80eec --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cassandra-sink.md @@ -0,0 +1,61 @@ +--- +id: io-cassandra-sink +title: Cassandra sink connector +sidebar_label: "Cassandra sink connector" +original_id: io-cassandra-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Cassandra sink connector pulls messages from Pulsar topics to Cassandra clusters. + +## Configuration + +The configuration of the Cassandra sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `roots` | String|true | " " (empty string) | A comma-separated list of Cassandra hosts to connect to.| +| `keyspace` | String|true| " " (empty string)| The key space used for writing pulsar messages.

    **Note: `keyspace` should be created prior to a Cassandra sink.**| +| `keyname` | String|true| " " (empty string)| The key name of the Cassandra column family.

    The column is used for storing Pulsar message keys.

    If a Pulsar message doesn't have any key associated, the message value is used as the key. | +| `columnFamily` | String|true| " " (empty string)| The Cassandra column family name.

    **Note: `columnFamily` should be created prior to a Cassandra sink.**| +| `columnName` | String|true| " " (empty string) | The column name of the Cassandra column family.

    The column is used for storing Pulsar message values. | + +### Example + +Before using the Cassandra sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "roots": "localhost:9042", + "keyspace": "pulsar_test_keyspace", + "columnFamily": "pulsar_test_table", + "keyname": "key", + "columnName": "col" + } + + ``` + +* YAML + + ``` + + configs: + roots: "localhost:9042" + keyspace: "pulsar_test_keyspace" + columnFamily: "pulsar_test_table" + keyname: "key" + columnName: "col" + + ``` + +## Usage + +For more information about **how to connect Pulsar with Cassandra**, see [here](io-quickstart.md#connect-pulsar-to-apache-cassandra). diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cdc-debezium.md b/site2/website-next/versioned_docs/version-2.7.2/io-cdc-debezium.md new file mode 100644 index 0000000000000..bc32077f562ae --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cdc-debezium.md @@ -0,0 +1,554 @@ +--- +id: io-cdc-debezium +title: Debezium source connector +sidebar_label: "Debezium source connector" +original_id: io-cdc-debezium +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Debezium source connector pulls messages from MySQL or PostgreSQL +and persists the messages to Pulsar topics. + +## Configuration + +The configuration of Debezium source connector has the following properties. + +| Name | Required | Default | Description | +|------|----------|---------|-------------| +| `task.class` | true | null | A source task class that implemented in Debezium. | +| `database.hostname` | true | null | The address of a database server. | +| `database.port` | true | null | The port number of a database server.| +| `database.user` | true | null | The name of a database user that has the required privileges. | +| `database.password` | true | null | The password for a database user that has the required privileges. | +| `database.server.id` | true | null | The connector’s identifier that must be unique within a database cluster and similar to the database’s server-id configuration property. | +| `database.server.name` | true | null | The logical name of a database server/cluster, which forms a namespace and it is used in all the names of Kafka topics to which the connector writes, the Kafka Connect schema names, and the namespaces of the corresponding Avro schema when the Avro Connector is used. | +| `database.whitelist` | false | null | A list of all databases hosted by this server which is monitored by the connector.

    This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring. | +| `key.converter` | true | null | The converter provided by Kafka Connect to convert record key. | +| `value.converter` | true | null | The converter provided by Kafka Connect to convert record value. | +| `database.history` | true | null | The name of the database history class. | +| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements.

    **Note: this topic is for internal use only and should not be used by consumers.** | +| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. | +| `pulsar.service.url` | true | null | Pulsar cluster service URL for the offset topic used in Debezium. You can use the `bin/pulsar-admin --admin-url http://pulsar:8080 sources localrun --source-config-file configs/pg-pulsar-config.yaml` command to point to the target Pulsar cluster. | +| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. | +| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). | +| `mongodb.name` | true | null | A unique name that identifies the connector and/or MongoDB replica set or shared cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster. | +| `mongodb.user` | true | null | Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. | +| `mongodb.password` | true | null | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. | +| `mongodb.task.id` | true | null | The taskId of the MongoDB connector that attempts to use a separate task for each replica set. | + + + +## Example of MySQL + +You need to create a configuration file before using the Pulsar Debezium connector. + +### Configuration + +You can use one of the following methods to create a configuration file. + +* JSON + + ```json + + { + "database.hostname": "localhost", + "database.port": "3306", + "database.user": "debezium", + "database.password": "dbz", + "database.server.id": "184054", + "database.server.name": "dbserver1", + "database.whitelist": "inventory", + "database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory", + "database.history.pulsar.topic": "history-topic", + "database.history.pulsar.service.url": "pulsar://127.0.0.1:6650", + "key.converter": "org.apache.kafka.connect.json.JsonConverter", + "value.converter": "org.apache.kafka.connect.json.JsonConverter", + "pulsar.service.url": "pulsar://127.0.0.1:6650", + "offset.storage.topic": "offset-topic" + } + + ``` + +* YAML + + You can create a `debezium-mysql-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/resources/debezium-mysql-source-config.yaml) below to the `debezium-mysql-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-mysql-source" + topicName: "debezium-mysql-topic" + archive: "connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for mysql, docker image: debezium/example-mysql:0.8 + database.hostname: "localhost" + database.port: "3306" + database.user: "debezium" + database.password: "dbz" + database.server.id: "184054" + database.server.name: "dbserver1" + database.whitelist: "inventory" + database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory" + database.history.pulsar.topic: "history-topic" + database.history.pulsar.service.url: "pulsar://127.0.0.1:6650" + + ## KEY_CONVERTER_CLASS_CONFIG, VALUE_CONVERTER_CLASS_CONFIG + key.converter: "org.apache.kafka.connect.json.JsonConverter" + value.converter: "org.apache.kafka.connect.json.JsonConverter" + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ## OFFSET_STORAGE_TOPIC_CONFIG + offset.storage.topic: "offset-topic" + + ``` + +### Usage + +This example shows how to change the data of a MySQL table using the Pulsar Debezium connector. + +1. Start a MySQL server with a database from which Debezium can capture changes. + + ```bash + + $ docker run -it --rm \ + --name mysql \ + -p 3306:3306 \ + -e MYSQL_ROOT_PASSWORD=debezium \ + -e MYSQL_USER=mysqluser \ + -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:0.8 + + ``` + +2. Start a Pulsar service locally in standalone mode. + + ```bash + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar`. + + ```bash + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar \ + --name debezium-mysql-source --destination-topic-name debezium-mysql-topic \ + --tenant public \ + --namespace default \ + --source-config '{"database.hostname": "localhost","database.port": "3306","database.user": "debezium","database.password": "dbz","database.server.id": "184054","database.server.name": "dbserver1","database.whitelist": "inventory","database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory","database.history.pulsar.topic": "history-topic","database.history.pulsar.service.url": "pulsar://127.0.0.1:6650","key.converter": "org.apache.kafka.connect.json.JsonConverter","value.converter": "org.apache.kafka.connect.json.JsonConverter","pulsar.service.url": "pulsar://127.0.0.1:6650","offset.storage.topic": "offset-topic"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ```bash + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-mysql-source-config.yaml + + ``` + +4. Subscribe the topic _sub-products_ for the table _inventory.products_. + + ```bash + + $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0 + + ``` + +5. Start a MySQL client in docker. + + ```bash + + $ docker run -it --rm \ + --name mysqlterm \ + --link mysql \ + --rm mysql:5.7 sh \ + -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"' + + ``` + +6. A MySQL client pops out. + + Use the following commands to change the data of the table _products_. + + ``` + + mysql> use inventory; + mysql> show tables; + mysql> SELECT * FROM products; + mysql> UPDATE products SET name='1111111111' WHERE id=101; + mysql> UPDATE products SET name='1111111111' WHERE id=107; + + ``` + + In the terminal window of subscribing topic, you can find the data changes have been kept in the _sub-products_ topic. + +## Example of PostgreSQL + +You need to create a configuration file before using the Pulsar Debezium connector. + +### Configuration + +You can use one of the following methods to create a configuration file. + +* JSON + + ```json + + { + "database.hostname": "localhost", + "database.port": "5432", + "database.user": "postgres", + "database.password": "postgres", + "database.dbname": "postgres", + "database.server.name": "dbserver1", + "schema.whitelist": "inventory", + "pulsar.service.url": "pulsar://127.0.0.1:6650" + } + + ``` + +* YAML + + You can create a `debezium-postgres-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/resources/debezium-postgres-source-config.yaml) below to the `debezium-postgres-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-postgres-source" + topicName: "debezium-postgres-topic" + archive: "connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for pg, docker image: debezium/example-postgress:0.8 + database.hostname: "localhost" + database.port: "5432" + database.user: "postgres" + database.password: "postgres" + database.dbname: "postgres" + database.server.name: "dbserver1" + schema.whitelist: "inventory" + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ``` + +### Usage + +This example shows how to change the data of a PostgreSQL table using the Pulsar Debezium connector. + + +1. Start a PostgreSQL server with a database from which Debezium can capture changes. + + ```bash + + $ docker pull debezium/example-postgres:0.8 + $ docker run -d -it --rm --name pulsar-postgresql -p 5432:5432 debezium/example-postgres:0.8 + + ``` + +2. Start a Pulsar service locally in standalone mode. + + ```bash + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar`. + + ```bash + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar \ + --name debezium-postgres-source \ + --destination-topic-name debezium-postgres-topic \ + --tenant public \ + --namespace default \ + --source-config '{"database.hostname": "localhost","database.port": "5432","database.user": "postgres","database.password": "postgres","database.dbname": "postgres","database.server.name": "dbserver1","schema.whitelist": "inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ```bash + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-postgres-source-config.yaml + + ``` + +4. Subscribe the topic _sub-products_ for the _inventory.products_ table. + + ``` + + $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0 + + ``` + +5. Start a PostgreSQL client in docker. + + ```bash + + $ docker exec -it pulsar-postgresql /bin/bash + + ``` + +6. A PostgreSQL client pops out. + + Use the following commands to change the data of the table _products_. + + ``` + + psql -U postgres postgres + postgres=# \c postgres; + You are now connected to database "postgres" as user "postgres". + postgres=# SET search_path TO inventory; + SET + postgres=# select * from products; + id | name | description | weight + -----+--------------------+---------------------------------------------------------+-------- + 102 | car battery | 12V car battery | 8.1 + 103 | 12-pack drill bits | 12-pack of drill bits with sizes ranging from #40 to #3 | 0.8 + 104 | hammer | 12oz carpenter's hammer | 0.75 + 105 | hammer | 14oz carpenter's hammer | 0.875 + 106 | hammer | 16oz carpenter's hammer | 1 + 107 | rocks | box of assorted rocks | 5.3 + 108 | jacket | water resistent black wind breaker | 0.1 + 109 | spare tire | 24 inch spare tire | 22.2 + 101 | 1111111111 | Small 2-wheel scooter | 3.14 + (9 rows) + + postgres=# UPDATE products SET name='1111111111' WHERE id=107; + UPDATE 1 + + ``` + + In the terminal window of subscribing topic, you can receive the following messages. + + ```bash + + ----- got message ----- + {"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":107}}�{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"db"},{"type":"int64","optional":true,"field":"ts_usec"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"string","optional":true,"field":"schema"},{"type":"string","optional":true,"field":"table"},{"type":"boolean","optional":true,"default":false,"field":"snapshot"},{"type":"boolean","optional":true,"field":"last_snapshot_record"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"before":{"id":107,"name":"rocks","description":"box of assorted rocks","weight":5.3},"after":{"id":107,"name":"1111111111","description":"box of assorted rocks","weight":5.3},"source":{"version":"0.9.2.Final","connector":"postgresql","name":"dbserver1","db":"postgres","ts_usec":1559208957661080,"txId":577,"lsn":23862872,"schema":"inventory","table":"products","snapshot":false,"last_snapshot_record":null},"op":"u","ts_ms":1559208957692}} + + ``` + +## Example of MongoDB + +You need to create a configuration file before using the Pulsar Debezium connector. + +* JSON + + ```json + + { + "mongodb.hosts": "rs0/mongodb:27017", + "mongodb.name": "dbserver1", + "mongodb.user": "debezium", + "mongodb.password": "dbz", + "mongodb.task.id": "1", + "database.whitelist": "inventory", + "pulsar.service.url": "pulsar://127.0.0.1:6650" + } + + ``` + +* YAML + + You can create a `debezium-mongodb-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mongodb/src/main/resources/debezium-mongodb-source-config.yaml) below to the `debezium-mongodb-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-mongodb-source" + topicName: "debezium-mongodb-topic" + archive: "connectors/pulsar-io-debezium-mongodb-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for pg, docker image: debezium/example-postgress:0.10 + mongodb.hosts: "rs0/mongodb:27017", + mongodb.name: "dbserver1", + mongodb.user: "debezium", + mongodb.password: "dbz", + mongodb.task.id: "1", + database.whitelist: "inventory", + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ``` + +### Usage + +This example shows how to change the data of a MongoDB table using the Pulsar Debezium connector. + + +1. Start a MongoDB server with a database from which Debezium can capture changes. + + ```bash + + $ docker pull debezium/example-mongodb:0.10 + $ docker run -d -it --rm --name pulsar-mongodb -e MONGODB_USER=mongodb -e MONGODB_PASSWORD=mongodb -p 27017:27017 debezium/example-mongodb:0.10 + + ``` + + Use the following commands to initialize the data. + + ``` bash + + ./usr/local/bin/init-inventory.sh + + ``` + + If the local host cannot access the container network, you can update the file ```/etc/hosts``` and add a rule ```127.0.0.1 6 f114527a95f```. f114527a95f is container id, you can try to get by ```docker ps -a``` + + +2. Start a Pulsar service locally in standalone mode. + + ``` + + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-mongodb-@pulsar:version@.nar`. + + ``` + + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-mongodb-@pulsar:version@.nar \ + --name debezium-mongodb-source \ + --destination-topic-name debezium-mongodb-topic \ + --tenant public \ + --namespace default \ + --source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user": "debezium","mongodb.password": "dbz","mongodb.task.id": "1","database.whitelist": "inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ``` + + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-mongodb-source-config.yaml + + ``` + +4. Subscribe the topic _sub-products_ for the _inventory.products_ table. + + ``` + + + $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0 + + ``` + +5. Start a MongoDB client in docker. + + ``` + + + $ docker exec -it pulsar-mongodb /bin/bash + + ``` + +6. A MongoDB client pops out. + + ``` + + + mongo -u debezium -p dbz --authenticationDatabase admin localhost:27017/inventory + db.products.update({"_id":NumberLong(104)},{$set:{weight:1.25}}) + + ``` + + In the terminal window of subscribing topic, you can receive the following messages. + + ``` + + + ----- got message ----- + {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":"104"}}, value = {"schema":{"type":"struct","fields":[{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"after"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"patch"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"field":"rs"},{"type":"string","optional":false,"field":"collection"},{"type":"int32","optional":false,"field":"ord"},{"type":"int64","optional":true,"field":"h"}],"optional":false,"name":"io.debezium.connector.mongo.Source","field":"source"},{"type":"string","optional":true,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"after":"{\"_id\": {\"$numberLong\": \"104\"},\"name\": \"hammer\",\"description\": \"12oz carpenter's hammer\",\"weight\": 1.25,\"quantity\": 4}","patch":null,"source":{"version":"0.10.0.Final","connector":"mongodb","name":"dbserver1","ts_ms":1573541905000,"snapshot":"true","db":"inventory","rs":"rs0","collection":"products","ord":1,"h":4983083486544392763},"op":"r","ts_ms":1573541909761}}. + + ``` + +## FAQ + +### Debezium postgres connector will hang when create snap + +``` + +#18 prio=5 os_prio=31 tid=0x00007fd83096f800 nid=0xa403 waiting on condition [0x000070000f534000] + java.lang.Thread.State: WAITING (parking) + at sun.misc.Unsafe.park(Native Method) + - parking to wait for <0x00000007ab025a58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) + at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) + at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) + at java.util.concurrent.LinkedBlockingDeque.putLast(LinkedBlockingDeque.java:396) + at java.util.concurrent.LinkedBlockingDeque.put(LinkedBlockingDeque.java:649) + at io.debezium.connector.base.ChangeEventQueue.enqueue(ChangeEventQueue.java:132) + at io.debezium.connector.postgresql.PostgresConnectorTask$Lambda$203/385424085.accept(Unknown Source) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.sendCurrentRecord(RecordsSnapshotProducer.java:402) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.readTable(RecordsSnapshotProducer.java:321) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.lambda$takeSnapshot$6(RecordsSnapshotProducer.java:226) + at io.debezium.connector.postgresql.RecordsSnapshotProducer$Lambda$240/1347039967.accept(Unknown Source) + at io.debezium.jdbc.JdbcConnection.queryWithBlockingConsumer(JdbcConnection.java:535) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.takeSnapshot(RecordsSnapshotProducer.java:224) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.lambda$start$0(RecordsSnapshotProducer.java:87) + at io.debezium.connector.postgresql.RecordsSnapshotProducer$Lambda$206/589332928.run(Unknown Source) + at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705) + at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:717) + at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2010) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.start(RecordsSnapshotProducer.java:87) + at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:126) + at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:47) + at org.apache.pulsar.io.kafka.connect.KafkaConnectSource.open(KafkaConnectSource.java:127) + at org.apache.pulsar.io.debezium.DebeziumSource.open(DebeziumSource.java:100) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:690) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:200) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:230) + at java.lang.Thread.run(Thread.java:748) + +``` + +If you encounter the above problems in synchronizing data, please refer to [this](https://github.com/apache/pulsar/issues/4075) and add the following configuration to the configuration file: + +``` + +max.queue.size= + +``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md b/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md index f78686012d45e..9ce8f7fc39101 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cdc.md @@ -1,7 +1,7 @@ --- id: io-cdc title: CDC connector -sidebar_label: CDC connector +sidebar_label: "CDC connector" original_id: io-cdc --- diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-cli.md b/site2/website-next/versioned_docs/version-2.7.2/io-cli.md index c9c19e4c3c492..81c3cd665e8c5 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-cli.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-cli.md @@ -1,7 +1,7 @@ --- id: io-cli title: Connector Admin CLI -sidebar_label: CLI +sidebar_label: "CLI" original_id: io-cli --- diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md b/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md index 47ee183c6367b..3e0924a7f3c91 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-connectors.md @@ -1,7 +1,7 @@ --- id: io-connectors title: Built-in connector -sidebar_label: Built-in connector +sidebar_label: "Built-in connector" original_id: io-connectors --- diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-debezium-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-debezium-source.md new file mode 100644 index 0000000000000..692d1d64e9ead --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-debezium-source.md @@ -0,0 +1,589 @@ +--- +id: io-debezium-source +title: Debezium source connector +sidebar_label: "Debezium source connector" +original_id: io-debezium-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Debezium source connector pulls messages from MySQL or PostgreSQL +and persists the messages to Pulsar topics. + +## Configuration + +The configuration of Debezium source connector has the following properties. + +| Name | Required | Default | Description | +|------|----------|---------|-------------| +| `task.class` | true | null | A source task class that implemented in Debezium. | +| `database.hostname` | true | null | The address of a database server. | +| `database.port` | true | null | The port number of a database server.| +| `database.user` | true | null | The name of a database user that has the required privileges. | +| `database.password` | true | null | The password for a database user that has the required privileges. | +| `database.server.id` | true | null | The connector’s identifier that must be unique within a database cluster and similar to the database’s server-id configuration property. | +| `database.server.name` | true | null | The logical name of a database server/cluster, which forms a namespace and it is used in all the names of Kafka topics to which the connector writes, the Kafka Connect schema names, and the namespaces of the corresponding Avro schema when the Avro Connector is used. | +| `database.whitelist` | false | null | A list of all databases hosted by this server which is monitored by the connector.

    This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring. | +| `key.converter` | true | null | The converter provided by Kafka Connect to convert record key. | +| `value.converter` | true | null | The converter provided by Kafka Connect to convert record value. | +| `database.history` | true | null | The name of the database history class. | +| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements.

    **Note: this topic is for internal use only and should not be used by consumers.** | +| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. | +| `pulsar.service.url` | true | null | Pulsar cluster service URL. | +| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. | +| `json-with-envelope` | false | false | Present the message only consist of payload. + +### Converter Options + +1. org.apache.kafka.connect.json.JsonConverter + +This config `json-with-envelope` is valid only for the JsonConverter. It's default value is false, the consumer use the schema ` +Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, +and the message only consist of payload. + +If the config `json-with-envelope` value is true, the consumer use the schema +`Schema.KeyValue(Schema.BYTES, Schema.BYTES`, the message consist of schema and payload. + +2. org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter + +If users select the AvroConverter, then the pulsar consumer should use the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), +Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consist of payload. + +### MongoDB Configuration +| Name | Required | Default | Description | +|------|----------|---------|-------------| +| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). | +| `mongodb.name` | true | null | A unique name that identifies the connector and/or MongoDB replica set or shared cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster. | +| `mongodb.user` | true | null | Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. | +| `mongodb.password` | true | null | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. | +| `mongodb.task.id` | true | null | The taskId of the MongoDB connector that attempts to use a separate task for each replica set. | + + + +## Example of MySQL + +You need to create a configuration file before using the Pulsar Debezium connector. + +### Configuration + +You can use one of the following methods to create a configuration file. + +* JSON + + ```json + + { + "database.hostname": "localhost", + "database.port": "3306", + "database.user": "debezium", + "database.password": "dbz", + "database.server.id": "184054", + "database.server.name": "dbserver1", + "database.whitelist": "inventory", + "database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory", + "database.history.pulsar.topic": "history-topic", + "database.history.pulsar.service.url": "pulsar://127.0.0.1:6650", + "key.converter": "org.apache.kafka.connect.json.JsonConverter", + "value.converter": "org.apache.kafka.connect.json.JsonConverter", + "pulsar.service.url": "pulsar://127.0.0.1:6650", + "offset.storage.topic": "offset-topic" + } + + ``` + +* YAML + + You can create a `debezium-mysql-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/resources/debezium-mysql-source-config.yaml) below to the `debezium-mysql-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-mysql-source" + topicName: "debezium-mysql-topic" + archive: "connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for mysql, docker image: debezium/example-mysql:0.8 + database.hostname: "localhost" + database.port: "3306" + database.user: "debezium" + database.password: "dbz" + database.server.id: "184054" + database.server.name: "dbserver1" + database.whitelist: "inventory" + database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory" + database.history.pulsar.topic: "history-topic" + database.history.pulsar.service.url: "pulsar://127.0.0.1:6650" + + ## KEY_CONVERTER_CLASS_CONFIG, VALUE_CONVERTER_CLASS_CONFIG + key.converter: "org.apache.kafka.connect.json.JsonConverter" + value.converter: "org.apache.kafka.connect.json.JsonConverter" + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ## OFFSET_STORAGE_TOPIC_CONFIG + offset.storage.topic: "offset-topic" + + ``` + +### Usage + +This example shows how to change the data of a MySQL table using the Pulsar Debezium connector. + +1. Start a MySQL server with a database from which Debezium can capture changes. + + ```bash + + $ docker run -it --rm \ + --name mysql \ + -p 3306:3306 \ + -e MYSQL_ROOT_PASSWORD=debezium \ + -e MYSQL_USER=mysqluser \ + -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:0.8 + + ``` + +2. Start a Pulsar service locally in standalone mode. + + ```bash + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar`. + + ```bash + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-mysql-@pulsar:version@.nar \ + --name debezium-mysql-source --destination-topic-name debezium-mysql-topic \ + --tenant public \ + --namespace default \ + --source-config '{"database.hostname": "localhost","database.port": "3306","database.user": "debezium","database.password": "dbz","database.server.id": "184054","database.server.name": "dbserver1","database.whitelist": "inventory","database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory","database.history.pulsar.topic": "history-topic","database.history.pulsar.service.url": "pulsar://127.0.0.1:6650","key.converter": "org.apache.kafka.connect.json.JsonConverter","value.converter": "org.apache.kafka.connect.json.JsonConverter","pulsar.service.url": "pulsar://127.0.0.1:6650","offset.storage.topic": "offset-topic"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ```bash + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-mysql-source-config.yaml + + ``` + +4. Subscribe the topic _sub-products_ for the table _inventory.products_. + + ```bash + + $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0 + + ``` + +5. Start a MySQL client in docker. + + ```bash + + $ docker run -it --rm \ + --name mysqlterm \ + --link mysql \ + --rm mysql:5.7 sh \ + -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"' + + ``` + +6. A MySQL client pops out. + + Use the following commands to change the data of the table _products_. + + ``` + + mysql> use inventory; + mysql> show tables; + mysql> SELECT * FROM products; + mysql> UPDATE products SET name='1111111111' WHERE id=101; + mysql> UPDATE products SET name='1111111111' WHERE id=107; + + ``` + + In the terminal window of subscribing topic, you can find the data changes have been kept in the _sub-products_ topic. + +## Example of PostgreSQL + +You need to create a configuration file before using the Pulsar Debezium connector. + +### Configuration + +You can use one of the following methods to create a configuration file. + +* JSON + + ```json + + { + "database.hostname": "localhost", + "database.port": "5432", + "database.user": "postgres", + "database.password": "changeme", + "database.dbname": "postgres", + "database.server.name": "dbserver1", + "plugin.name": "pgoutput", + "schema.whitelist": "public", + "table.whitelist": "public.users", + "pulsar.service.url": "pulsar://127.0.0.1:6650" + } + + ``` + +* YAML + + You can create a `debezium-postgres-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/resources/debezium-postgres-source-config.yaml) below to the `debezium-postgres-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-postgres-source" + topicName: "debezium-postgres-topic" + archive: "connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for postgres version 10+, official docker image: postgres:<10+> + database.hostname: "localhost" + database.port: "5432" + database.user: "postgres" + database.password: "changeme" + database.dbname: "postgres" + database.server.name: "dbserver1" + plugin.name: "pgoutput" + schema.whitelist: "public" + table.whitelist: "public.users" + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ``` + +Notice that `pgoutput` is a standard plugin of Postgres introduced in version 10 - [see Postgres architecture docu](https://www.postgresql.org/docs/10/logical-replication-architecture.html). You don't need to install anything, just make sure the WAL level is set to `logical` (see docker command below and [Postgres docu](https://www.postgresql.org/docs/current/runtime-config-wal.html)). + +### Usage + +This example shows how to change the data of a PostgreSQL table using the Pulsar Debezium connector. + + +1. Start a PostgreSQL server with a database from which Debezium can capture changes. + + ```bash + + $ docker run -d -it --rm \ + --name pulsar-postgres \ + -p 5432:5432 \ + -e POSTGRES_PASSWORD=changeme \ + postgres:13.3 -c wal_level=logical + + ``` + +2. Start a Pulsar service locally in standalone mode. + + ```bash + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar`. + + ```bash + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-postgres-@pulsar:version@.nar \ + --name debezium-postgres-source \ + --destination-topic-name debezium-postgres-topic \ + --tenant public \ + --namespace default \ + --source-config '{"database.hostname": "localhost","database.port": "5432","database.user": "postgres","database.password": "changeme","database.dbname": "postgres","database.server.name": "dbserver1","schema.whitelist": "public","table.whitelist": "public.users","pulsar.service.url": "pulsar://127.0.0.1:6650"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ```bash + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-postgres-source-config.yaml + + ``` + +4. Subscribe the topic _sub-users_ for the _public.users_ table. + + ``` + + $ bin/pulsar-client consume -s "sub-users" public/default/dbserver1.public.users -n 0 + + ``` + +5. Start a PostgreSQL client in docker. + + ```bash + + $ docker exec -it pulsar-postgresql /bin/bash + + ``` + +6. A PostgreSQL client pops out. + + Use the following commands to create sample data in the table _users_. + + ``` + + psql -U postgres -h localhost -p 5432 + Password for user postgres: + + CREATE TABLE users( + id BIGINT GENERATED ALWAYS AS IDENTITY, PRIMARY KEY(id), + hash_firstname TEXT NOT NULL, + hash_lastname TEXT NOT NULL, + gender VARCHAR(6) NOT NULL CHECK (gender IN ('male', 'female')) + ); + + INSERT INTO users(hash_firstname, hash_lastname, gender) + SELECT md5(RANDOM()::TEXT), md5(RANDOM()::TEXT), CASE WHEN RANDOM() < 0.5 THEN 'male' ELSE 'female' END FROM generate_series(1, 100); + + postgres=# select * from users; + + id | hash_firstname | hash_lastname | gender + -------+----------------------------------+----------------------------------+-------- + 1 | 02bf7880eb489edc624ba637f5ab42bd | 3e742c2cc4217d8e3382cc251415b2fb | female + 2 | dd07064326bb9119189032316158f064 | 9c0e938f9eddbd5200ba348965afbc61 | male + 3 | 2c5316fdd9d6595c1cceb70eed12e80c | 8a93d7d8f9d76acfaaa625c82a03ea8b | female + 4 | 3dfa3b4f70d8cd2155567210e5043d2b | 32c156bc28f7f03ab5d28e2588a3dc19 | female + + + postgres=# UPDATE users SET hash_firstname='maxim' WHERE id=1; + UPDATE 1 + + ``` + + In the terminal window of subscribing topic, you can receive the following messages. + + ```bash + + ----- got message ----- + {"before":null,"after":{"id":1,"hash_firstname":"maxim","hash_lastname":"292113d30a3ccee0e19733dd7f88b258","gender":"male"},"source:{"version":"1.0.0.Final","connector":"postgresql","name":"foobar","ts_ms":1624045862644,"snapshot":"false","db":"postgres","schema":"public","table":"users","txId":595,"lsn":24419784,"xmin":null},"op":"u","ts_ms":1624045862648} + ...many more + + ``` + +## Example of MongoDB + +You need to create a configuration file before using the Pulsar Debezium connector. + +* JSON + + ```json + + { + "mongodb.hosts": "rs0/mongodb:27017", + "mongodb.name": "dbserver1", + "mongodb.user": "debezium", + "mongodb.password": "dbz", + "mongodb.task.id": "1", + "database.whitelist": "inventory", + "pulsar.service.url": "pulsar://127.0.0.1:6650" + } + + ``` + +* YAML + + You can create a `debezium-mongodb-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mongodb/src/main/resources/debezium-mongodb-source-config.yaml) below to the `debezium-mongodb-source-config.yaml` file. + + ```yaml + + tenant: "public" + namespace: "default" + name: "debezium-mongodb-source" + topicName: "debezium-mongodb-topic" + archive: "connectors/pulsar-io-debezium-mongodb-@pulsar:version@.nar" + parallelism: 1 + + configs: + + ## config for pg, docker image: debezium/example-mongodb:0.10 + mongodb.hosts: "rs0/mongodb:27017", + mongodb.name: "dbserver1", + mongodb.user: "debezium", + mongodb.password: "dbz", + mongodb.task.id: "1", + database.whitelist: "inventory", + + ## PULSAR_SERVICE_URL_CONFIG + pulsar.service.url: "pulsar://127.0.0.1:6650" + + ``` + +### Usage + +This example shows how to change the data of a MongoDB table using the Pulsar Debezium connector. + + +1. Start a MongoDB server with a database from which Debezium can capture changes. + + ```bash + + $ docker pull debezium/example-mongodb:0.10 + $ docker run -d -it --rm --name pulsar-mongodb -e MONGODB_USER=mongodb -e MONGODB_PASSWORD=mongodb -p 27017:27017 debezium/example-mongodb:0.10 + + ``` + + Use the following commands to initialize the data. + + ``` bash + + ./usr/local/bin/init-inventory.sh + + ``` + + If the local host cannot access the container network, you can update the file ```/etc/hosts``` and add a rule ```127.0.0.1 6 f114527a95f```. f114527a95f is container id, you can try to get by ```docker ps -a``` + + +2. Start a Pulsar service locally in standalone mode. + + ``` + + + $ bin/pulsar standalone + + ``` + +3. Start the Pulsar Debezium connector in local run mode using one of the following methods. + + * Use the **JSON** configuration file as shown previously. + + Make sure the nar file is available at `connectors/pulsar-io-mongodb-@pulsar:version@.nar`. + + ``` + + + $ bin/pulsar-admin source localrun \ + --archive connectors/pulsar-io-debezium-mongodb-@pulsar:version@.nar \ + --name debezium-mongodb-source \ + --destination-topic-name debezium-mongodb-topic \ + --tenant public \ + --namespace default \ + --source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user": "debezium","mongodb.password": "dbz","mongodb.task.id": "1","database.whitelist": "inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}' + + ``` + + * Use the **YAML** configuration file as shown previously. + + ``` + + + $ bin/pulsar-admin source localrun \ + --source-config-file debezium-mongodb-source-config.yaml + + ``` + +4. Subscribe the topic _sub-products_ for the _inventory.products_ table. + + ``` + + + $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0 + + ``` + +5. Start a MongoDB client in docker. + + ``` + + + $ docker exec -it pulsar-mongodb /bin/bash + + ``` + +6. A MongoDB client pops out. + + ``` + + + mongo -u debezium -p dbz --authenticationDatabase admin localhost:27017/inventory + db.products.update({"_id":NumberLong(104)},{$set:{weight:1.25}}) + + ``` + + In the terminal window of subscribing topic, you can receive the following messages. + + ``` + + + ----- got message ----- + {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":"104"}}, value = {"schema":{"type":"struct","fields":[{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"after"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"patch"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"field":"rs"},{"type":"string","optional":false,"field":"collection"},{"type":"int32","optional":false,"field":"ord"},{"type":"int64","optional":true,"field":"h"}],"optional":false,"name":"io.debezium.connector.mongo.Source","field":"source"},{"type":"string","optional":true,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"after":"{\"_id\": {\"$numberLong\": \"104\"},\"name\": \"hammer\",\"description\": \"12oz carpenter's hammer\",\"weight\": 1.25,\"quantity\": 4}","patch":null,"source":{"version":"0.10.0.Final","connector":"mongodb","name":"dbserver1","ts_ms":1573541905000,"snapshot":"true","db":"inventory","rs":"rs0","collection":"products","ord":1,"h":4983083486544392763},"op":"r","ts_ms":1573541909761}}. + + ``` + +## FAQ + +### Debezium postgres connector will hang when create snap + +``` + +#18 prio=5 os_prio=31 tid=0x00007fd83096f800 nid=0xa403 waiting on condition [0x000070000f534000] + java.lang.Thread.State: WAITING (parking) + at sun.misc.Unsafe.park(Native Method) + - parking to wait for <0x00000007ab025a58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) + at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) + at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) + at java.util.concurrent.LinkedBlockingDeque.putLast(LinkedBlockingDeque.java:396) + at java.util.concurrent.LinkedBlockingDeque.put(LinkedBlockingDeque.java:649) + at io.debezium.connector.base.ChangeEventQueue.enqueue(ChangeEventQueue.java:132) + at io.debezium.connector.postgresql.PostgresConnectorTask$Lambda$203/385424085.accept(Unknown Source) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.sendCurrentRecord(RecordsSnapshotProducer.java:402) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.readTable(RecordsSnapshotProducer.java:321) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.lambda$takeSnapshot$6(RecordsSnapshotProducer.java:226) + at io.debezium.connector.postgresql.RecordsSnapshotProducer$Lambda$240/1347039967.accept(Unknown Source) + at io.debezium.jdbc.JdbcConnection.queryWithBlockingConsumer(JdbcConnection.java:535) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.takeSnapshot(RecordsSnapshotProducer.java:224) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.lambda$start$0(RecordsSnapshotProducer.java:87) + at io.debezium.connector.postgresql.RecordsSnapshotProducer$Lambda$206/589332928.run(Unknown Source) + at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705) + at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:717) + at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2010) + at io.debezium.connector.postgresql.RecordsSnapshotProducer.start(RecordsSnapshotProducer.java:87) + at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:126) + at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:47) + at org.apache.pulsar.io.kafka.connect.KafkaConnectSource.open(KafkaConnectSource.java:127) + at org.apache.pulsar.io.debezium.DebeziumSource.open(DebeziumSource.java:100) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:690) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:200) + at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:230) + at java.lang.Thread.run(Thread.java:748) + +``` + +If you encounter the above problems in synchronizing data, please refer to [this](https://github.com/apache/pulsar/issues/4075) and add the following configuration to the configuration file: + +``` + +max.queue.size= + +``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-debug.md b/site2/website-next/versioned_docs/version-2.7.2/io-debug.md index fada2555acf71..f815e862cae42 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-debug.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-debug.md @@ -1,7 +1,7 @@ --- id: io-debug title: How to debug Pulsar connectors -sidebar_label: Debug +sidebar_label: "Debug" original_id: io-debug --- @@ -14,61 +14,66 @@ To better demonstrate how to debug Pulsar connectors, here takes a Mongo sink co **Deploy a Mongo sink environment** 1. Start a Mongo service. - ```bash - - docker pull mongo:4 - docker run -d -p 27017:27017 --name pulsar-mongo -v $PWD/data:/data/db mongo:4 + ```bash + + docker pull mongo:4 + docker run -d -p 27017:27017 --name pulsar-mongo -v $PWD/data:/data/db mongo:4 + + ``` - ``` 2. Create a DB and a collection. - ```bash - - docker exec -it pulsar-mongo /bin/bash - mongo - > use pulsar - > db.createCollection('messages') - > exit + ```bash + + docker exec -it pulsar-mongo /bin/bash + mongo + > use pulsar + > db.createCollection('messages') + > exit + + ``` - ``` 3. Start Pulsar standalone. - ```bash - - docker pull apachepulsar/pulsar:2.4.0 - docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --link pulsar-mongo --name pulsar-mongo-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone + ```bash + + docker pull apachepulsar/pulsar:2.4.0 + docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --link pulsar-mongo --name pulsar-mongo-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone + + ``` - ``` 4. Configure the Mongo sink with the `mongo-sink-config.yaml` file. - ```bash - - configs: - mongoUri: "mongodb://pulsar-mongo:27017" - database: "pulsar" - collection: "messages" - batchSize: 2 - batchTimeMs: 500 - - ``` - ```bash + ```bash + + configs: + mongoUri: "mongodb://pulsar-mongo:27017" + database: "pulsar" + collection: "messages" + batchSize: 2 + batchTimeMs: 500 + + ``` - docker cp mongo-sink-config.yaml pulsar-mongo-standalone:/pulsar/ + ```bash + + docker cp mongo-sink-config.yaml pulsar-mongo-standalone:/pulsar/ + + ``` - ``` 5. Download the Mongo sink nar package. - ```bash - - docker exec -it pulsar-mongo-standalone /bin/bash - curl -O http://apache.01link.hk/pulsar/pulsar-2.4.0/connectors/pulsar-io-mongo-2.4.0.nar + ```bash + + docker exec -it pulsar-mongo-standalone /bin/bash + curl -O http://apache.01link.hk/pulsar/pulsar-2.4.0/connectors/pulsar-io-mongo-2.4.0.nar + + ``` - ``` ## Debug in localrun mode Start the Mongo sink in localrun mode using the `localrun` command. :::tip - For more information about the `localrun` command, see [`localrun`](reference-connector-admin.md/#localrun-1). ::: @@ -84,138 +89,144 @@ For more information about the `localrun` command, see [`localrun`](reference-co --parallelism 1 ``` + ### Use connector log Use one of the following methods to get a connector log in localrun mode: * After executing the `localrun` command, the **log is automatically printed on the console**. * The log is located at: - - - ```bash - logs/functions/tenant/namespace/function-name/function-name-instance-id.log - - ``` - - **Example** - - The path of the Mongo sink connector is: + ```bash + + logs/functions/tenant/namespace/function-name/function-name-instance-id.log + + ``` - ```bash + **Example** + + The path of the Mongo sink connector is: - logs/functions/public/default/pulsar-mongo-sink/pulsar-mongo-sink-0.log + ```bash + + logs/functions/public/default/pulsar-mongo-sink/pulsar-mongo-sink-0.log + + ``` - ``` To clearly explain the log information, here breaks down the large block of information into small blocks and add descriptions for each block. * This piece of log information shows the storage path of the nar package after decompression. - ``` - 08:21:54.132 [main] INFO org.apache.pulsar.common.nar.NarClassLoader - Created class loader with paths: [file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/, file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/, + ``` + + 08:21:54.132 [main] INFO org.apache.pulsar.common.nar.NarClassLoader - Created class loader with paths: [file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/, file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/, + + ``` + + :::tip - ``` - -:::tip + If `class cannot be found` exception is thrown, check whether the nar file is decompressed in the folder `file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/` or not. -If `class cannot be found` exception is thrown, check whether the nar file is decompressed in the folder `file:/tmp/pulsar-nar/pulsar-io-mongo-2.4.0.nar-unpacked/META-INF/bundled-dependencies/` or not. + ::: -::: * This piece of log information illustrates the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, resources, and so on, which can be used to **check whether the Mongo sink connector is configured correctly or not**. - ```bash - - 08:21:55.390 [main] INFO org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=853d60a1-0c48-44d5-9a5c-6917386476b2, functionVersion=c2ce1458-b69e-4175-88c0-a0a856a2be8c, functionDetails=tenant: "public" - namespace: "default" - name: "pulsar-mongo-sink" - className: "org.apache.pulsar.functions.api.utils.IdentityFunction" - autoAck: true - parallelism: 1 - source { - typeClassName: "[B" - inputSpecs { - key: "test-mongo" - value { - } - } - cleanupSubscription: true - } - sink { - className: "org.apache.pulsar.io.mongodb.MongoSink" - configs: "{\"mongoUri\":\"mongodb://pulsar-mongo:27017\",\"database\":\"pulsar\",\"collection\":\"messages\",\"batchSize\":2,\"batchTimeMs\":500}" - typeClassName: "[B" - } - resources { - cpu: 1.0 - ram: 1073741824 - disk: 10737418240 - } - componentType: SINK - , maxBufferedTuples=1024, functionAuthenticationSpec=null, port=38459, clusterName=local) + ```bash + + 08:21:55.390 [main] INFO org.apache.pulsar.functions.runtime.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=853d60a1-0c48-44d5-9a5c-6917386476b2, functionVersion=c2ce1458-b69e-4175-88c0-a0a856a2be8c, functionDetails=tenant: "public" + namespace: "default" + name: "pulsar-mongo-sink" + className: "org.apache.pulsar.functions.api.utils.IdentityFunction" + autoAck: true + parallelism: 1 + source { + typeClassName: "[B" + inputSpecs { + key: "test-mongo" + value { + } + } + cleanupSubscription: true + } + sink { + className: "org.apache.pulsar.io.mongodb.MongoSink" + configs: "{\"mongoUri\":\"mongodb://pulsar-mongo:27017\",\"database\":\"pulsar\",\"collection\":\"messages\",\"batchSize\":2,\"batchTimeMs\":500}" + typeClassName: "[B" + } + resources { + cpu: 1.0 + ram: 1073741824 + disk: 10737418240 + } + componentType: SINK + , maxBufferedTuples=1024, functionAuthenticationSpec=null, port=38459, clusterName=local) + + ``` - ``` * This piece of log information demonstrates the status of the connections to Mongo and configuration information. - ```bash - - 08:21:56.231 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to pulsar-mongo:27017 - 08:21:56.326 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=pulsar-mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 0]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=89058800} + ```bash + + 08:21:56.231 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:8}] to pulsar-mongo:27017 + 08:21:56.326 [cluster-ClusterId{value='5d6396a3c9e77c0569ff00eb', description='null'}-pulsar-mongo:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=pulsar-mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 0]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=89058800} + + ``` - ``` * This piece of log information explains the configuration of consumers and clients, including the topic name, subscription name, subscription type, and so on. - ```bash - - 08:21:56.719 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: { - "topicNames" : [ "test-mongo" ], - "topicsPattern" : null, - "subscriptionName" : "public/default/pulsar-mongo-sink", - "subscriptionType" : "Shared", - "receiverQueueSize" : 1000, - "acknowledgementsGroupTimeMicros" : 100000, - "negativeAckRedeliveryDelayMicros" : 60000000, - "maxTotalReceiverQueueSizeAcrossPartitions" : 50000, - "consumerName" : null, - "ackTimeoutMillis" : 0, - "tickDurationMillis" : 1000, - "priorityLevel" : 0, - "cryptoFailureAction" : "CONSUME", - "properties" : { - "application" : "pulsar-sink", - "id" : "public/default/pulsar-mongo-sink", - "instance_id" : "0" - }, - "readCompacted" : false, - "subscriptionInitialPosition" : "Latest", - "patternAutoDiscoveryPeriod" : 1, - "regexSubscriptionMode" : "PersistentOnly", - "deadLetterPolicy" : null, - "autoUpdatePartitions" : true, - "replicateSubscriptionState" : false, - "resetIncludeHead" : false - } - 08:21:56.726 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: { - "serviceUrl" : "pulsar://localhost:6650", - "authPluginClassName" : null, - "authParams" : null, - "operationTimeoutMs" : 30000, - "statsIntervalSeconds" : 60, - "numIoThreads" : 1, - "numListenerThreads" : 1, - "connectionsPerBroker" : 1, - "useTcpNoDelay" : true, - "useTls" : false, - "tlsTrustCertsFilePath" : null, - "tlsAllowInsecureConnection" : false, - "tlsHostnameVerificationEnable" : false, - "concurrentLookupRequest" : 5000, - "maxLookupRequest" : 50000, - "maxNumberOfRejectedRequestPerConnection" : 50, - "keepAliveIntervalSeconds" : 30, - "connectionTimeoutMs" : 10000, - "requestTimeoutMs" : 60000, - "defaultBackoffIntervalNanos" : 100000000, - "maxBackoffIntervalNanos" : 30000000000 - } + ```bash + + 08:21:56.719 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: { + "topicNames" : [ "test-mongo" ], + "topicsPattern" : null, + "subscriptionName" : "public/default/pulsar-mongo-sink", + "subscriptionType" : "Shared", + "receiverQueueSize" : 1000, + "acknowledgementsGroupTimeMicros" : 100000, + "negativeAckRedeliveryDelayMicros" : 60000000, + "maxTotalReceiverQueueSizeAcrossPartitions" : 50000, + "consumerName" : null, + "ackTimeoutMillis" : 0, + "tickDurationMillis" : 1000, + "priorityLevel" : 0, + "cryptoFailureAction" : "CONSUME", + "properties" : { + "application" : "pulsar-sink", + "id" : "public/default/pulsar-mongo-sink", + "instance_id" : "0" + }, + "readCompacted" : false, + "subscriptionInitialPosition" : "Latest", + "patternAutoDiscoveryPeriod" : 1, + "regexSubscriptionMode" : "PersistentOnly", + "deadLetterPolicy" : null, + "autoUpdatePartitions" : true, + "replicateSubscriptionState" : false, + "resetIncludeHead" : false + } + 08:21:56.726 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: { + "serviceUrl" : "pulsar://localhost:6650", + "authPluginClassName" : null, + "authParams" : null, + "operationTimeoutMs" : 30000, + "statsIntervalSeconds" : 60, + "numIoThreads" : 1, + "numListenerThreads" : 1, + "connectionsPerBroker" : 1, + "useTcpNoDelay" : true, + "useTls" : false, + "tlsTrustCertsFilePath" : null, + "tlsAllowInsecureConnection" : false, + "tlsHostnameVerificationEnable" : false, + "concurrentLookupRequest" : 5000, + "maxLookupRequest" : 50000, + "maxNumberOfRejectedRequestPerConnection" : 50, + "keepAliveIntervalSeconds" : 30, + "connectionTimeoutMs" : 10000, + "requestTimeoutMs" : 60000, + "defaultBackoffIntervalNanos" : 100000000, + "maxBackoffIntervalNanos" : 30000000000 + } + + ``` - ``` ## Debug in cluster mode You can use the following methods to debug a connector in cluster mode: * [Use connector log](#use-connector-log) @@ -243,6 +254,7 @@ Pulsar admin CLI helps you debug Pulsar connectors with the following subcommand --parallelism 1 ``` + ### `get` Use the `get` command to get the basic information about the Mongo sink connector, such as tenant, namespace, name, parallelism, and so on. @@ -276,10 +288,10 @@ Use the `get` command to get the basic information about the Mongo sink connecto :::tip - For more information about the `get` command, see [`get`](reference-connector-admin.md/#get-1). ::: + ### `status` Use the `status` command to get the current status about the Mongo sink connector, such as the number of instance, the number of running instance, instanceId, workerId and so on. @@ -311,14 +323,14 @@ Use the `status` command to get the current status about the Mongo sink connecto } ``` -:::tip +:::tip For more information about the `status` command, see [`status`](reference-connector-admin.md/#stauts-1). - If there are multiple connectors running on a worker, `workerId` can locate the worker on which the specified connector is running. ::: + ### `topics stats` Use the `topics stats` command to get the stats for a topic and its connected producer and consumer, such as whether the topic has received messages or not, whether there is a backlog of messages or not, the available permits and other key information. All rates are computed over a 1-minute window and are relative to the last completed 1-minute period. @@ -369,12 +381,13 @@ Use the `topics stats` command to get the stats for a topic and its connected pr } ``` -:::tip +:::tip For more information about the `topic stats` command, see [`topic stats`](http://pulsar.apache.org/docs/en/pulsar-admin/#stats-1). ::: + ## Checklist This checklist indicates the major areas to check when you debug connectors. It is a reminder of what to look for to ensure a thorough review and an evaluation tool to get the status of connectors. * Does Pulsar start successfully? diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-develop.md b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md index df45d9324edd3..d7531a0d30d67 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-develop.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md @@ -1,7 +1,7 @@ --- id: io-develop title: How to develop Pulsar connectors -sidebar_label: Develop +sidebar_label: "Develop" original_id: io-develop --- @@ -33,70 +33,69 @@ interface, which means you need to implement the {@inject: github:open:/pulsar-i 1. Implement the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method. - ```java + ```java + + /** + * Open connector with configuration + * + * @param config initialization config + * @param sourceContext + * @throws Exception IO type exceptions when opening a connector + */ + void open(final Map config, SourceContext sourceContext) throws Exception; + + ``` - /** - * Open connector with configuration - * - * @param config initialization config - * @param sourceContext - * @throws Exception IO type exceptions when opening a connector - */ - void open(final Map config, SourceContext sourceContext) throws Exception; + This method is called when the source connector is initialized. - ``` + In this method, you can retrieve all connector specific settings through the passed-in `config` parameter and initialize all necessary resources. + + For example, a Kafka connector can create a Kafka client in this `open` method. - This method is called when the source connector is initialized. - - In this method, you can retrieve all connector specific settings through the passed-in `config` parameter and initialize all necessary resources. - - For example, a Kafka connector can create a Kafka client in this `open` method. - - Besides, Pulsar runtime also provides a `SourceContext` for the - connector to access runtime resources for tasks like collecting metrics. The implementation can save the `SourceContext` for future use. + Besides, Pulsar runtime also provides a `SourceContext` for the + connector to access runtime resources for tasks like collecting metrics. The implementation can save the `SourceContext` for future use. 2. Implement the {@inject: github:read:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java} method. - ```java - - /** - * Reads the next message from source. - * If source does not have any new messages, this call should block. - * @return next message from source. The return result should never be null - * @throws Exception - */ - Record read() throws Exception; - - ``` - - If nothing to return, the implementation should be blocking rather than returning `null`. - - The returned {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should encapsulate the following information, which is needed by Pulsar IO runtime. - - * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following variables: - - |Variable|Required|Description - |---|---|--- - `TopicName`|No|Pulsar topic name from which the record is originated from. - `Key`|No| Messages can optionally be tagged with keys.

    For more information, see [Routing modes](concepts-messaging.md#routing-modes).| - `Value`|Yes|Actual data of the record. - `EventTime`|No|Event time of the record from the source. - `PartitionId`|No| If the record is originated from a partitioned source, it returns its `PartitionId`.

    `PartitionId` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. - `RecordSequence`|No|If the record is originated from a sequential source, it returns its `RecordSequence`.

    `RecordSequence` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. - `Properties` |No| If the record carries user-defined properties, it returns those properties. - `DestinationTopic`|No|Topic to which message should be written. - `Message`|No|A class which carries data sent by users.

    For more information, see [Message.java](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java).| - - * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following methods: - - Method|Description - |---|--- - `ack` |Acknowledge that the record is fully processed. - `fail`|Indicate that the record fails to be processed. + ```java + + /** + * Reads the next message from source. + * If source does not have any new messages, this call should block. + * @return next message from source. The return result should never be null + * @throws Exception + */ + Record read() throws Exception; + + ``` + + If nothing to return, the implementation should be blocking rather than returning `null`. + + The returned {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should encapsulate the following information, which is needed by Pulsar IO runtime. + + * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following variables: + + |Variable|Required|Description + |---|---|--- + `TopicName`|No|Pulsar topic name from which the record is originated from. + `Key`|No| Messages can optionally be tagged with keys.

    For more information, see [Routing modes](concepts-messaging.md#routing-modes).| + `Value`|Yes|Actual data of the record. + `EventTime`|No|Event time of the record from the source. + `PartitionId`|No| If the record is originated from a partitioned source, it returns its `PartitionId`.

    `PartitionId` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. + `RecordSequence`|No|If the record is originated from a sequential source, it returns its `RecordSequence`.

    `RecordSequence` is used as a part of the unique identifier by Pulsar IO runtime to deduplicate messages and achieve exactly-once processing guarantee. + `Properties` |No| If the record carries user-defined properties, it returns those properties. + `DestinationTopic`|No|Topic to which message should be written. + `Message`|No|A class which carries data sent by users.

    For more information, see [Message.java](https://github.com/apache/pulsar/blob/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java).| + + * {@inject: github:Record:/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java} should provide the following methods: + + Method|Description + |---|--- + `ack` |Acknowledge that the record is fully processed. + `fail`|Indicate that the record fails to be processed. :::tip - For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. ::: @@ -107,37 +106,37 @@ Developing a sink connector **is similar to** developing a source connector, tha 1. Implement the {@inject: github:open:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method. - ```java - - /** - * Open connector with configuration - * - * @param config initialization config - * @param sinkContext - * @throws Exception IO type exceptions when opening a connector - */ - void open(final Map config, SinkContext sinkContext) throws Exception; - - ``` + ```java + + /** + * Open connector with configuration + * + * @param config initialization config + * @param sinkContext + * @throws Exception IO type exceptions when opening a connector + */ + void open(final Map config, SinkContext sinkContext) throws Exception; + + ``` 2. Implement the {@inject: github:write:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java} method. - ```java - - /** - * Write a message to Sink - * @param record record to write to sink - * @throws Exception - */ - void write(Record record) throws Exception; + ```java + + /** + * Write a message to Sink + * @param record record to write to sink + * @throws Exception + */ + void write(Record record) throws Exception; + + ``` - ``` + During the implementation, you can decide how to write the `Value` and + the `Key` to the actual source, and leverage all the provided information such as + `PartitionId` and `RecordSequence` to achieve different processing guarantees. - During the implementation, you can decide how to write the `Value` and - the `Key` to the actual source, and leverage all the provided information such as - `PartitionId` and `RecordSequence` to achieve different processing guarantees. - - You also need to ack records (if messages are sent successfully) or fail records (if messages fail to send). + You also need to ack records (if messages are sent successfully) or fail records (if messages fail to send). ## Test @@ -161,7 +160,6 @@ Pulsar uses [testcontainers](https://www.testcontainers.org/) **for all integrat :::tip - For more information about **how to create integration tests for Pulsar connectors**, see {@inject: github:IntegrationTests:/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io}. ::: @@ -176,10 +174,10 @@ work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-j :::note - If you plan to package and distribute your connector for others to use, you are obligated to ::: + license and copyright your own code properly. Remember to add the license and copyright to all libraries your code uses and to your distribution. > @@ -194,7 +192,6 @@ a bit of Java ClassLoader isolation. :::tip - For more information about **how NAR works**, see [here](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd). ::: @@ -232,7 +229,6 @@ For Gradle users, there is a [Gradle Nar plugin available on the Gradle Plugin P :::tip - For more information about an **how to use NAR for Pulsar connectors**, see {@inject: github:TwitterFirehose:/pulsar-io/twitter/pom.xml}. ::: @@ -268,3 +264,4 @@ You can use [maven-shade-plugin](https://maven.apache.org/plugins/maven-shade-pl ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-dynamodb-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-dynamodb-source.md new file mode 100644 index 0000000000000..4a93683eb4a5b --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-dynamodb-source.md @@ -0,0 +1,84 @@ +--- +id: io-dynamodb-source +title: AWS DynamoDB source connector +sidebar_label: "AWS DynamoDB source connector" +original_id: io-dynamodb-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The DynamoDB source connector pulls data from DynamoDB table streams and persists data into Pulsar. + +This connector uses the [DynamoDB Streams Kinesis Adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter), +which uses the [Kinesis Consumer Library](https://github.com/awslabs/amazon-kinesis-client) (KCL) to do the actual +consuming of messages. The KCL uses DynamoDB to track state for consumers and requires cloudwatch access to log metrics. + + +## Configuration + +The configuration of the DynamoDB source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +`initialPositionInStream`|InitialPositionInStream|false|LATEST|The position where the connector starts from.

    Below are the available options:

  • `AT_TIMESTAMP`: start from the record at or after the specified timestamp.

  • `LATEST`: start after the most recent data record.

  • `TRIM_HORIZON`: start from the oldest available data record.
    • +`startAtTime`|Date|false|" " (empty string)|If set to `AT_TIMESTAMP`, it specifies the point in time to start consumption. +`applicationName`|String|false|Pulsar IO connector|The name of the KCL application. Must be unique, as it is used to define the table name for the dynamo table used for state tracking.

    By default, the application name is included in the user agent string used to make AWS requests. This can assist with troubleshooting, for example, distinguish requests made by separate connector instances. +`checkpointInterval`|long|false|60000|The frequency of the KCL checkpoint in milliseconds. +`backoffTime`|long|false|3000|The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds. +`numRetries`|int|false|3|The number of re-attempts when the connector encounters an exception while trying to set a checkpoint. +`receiveQueueSize`|int|false|1000|The maximum number of AWS records that can be buffered inside the connector.

    Once the `receiveQueueSize` is reached, the connector does not consume any messages from Kinesis until some messages in the queue are successfully consumed. +`dynamoEndpoint`|String|false|" " (empty string)|The Dynamo end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`cloudwatchEndpoint`|String|false|" " (empty string)|The Cloudwatch end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`awsEndpoint`|String|false|" " (empty string)|The DynamoDB Streams end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`awsRegion`|String|false|" " (empty string)|The AWS region.

    **Example**
    us-west-1, us-west-2 +`awsDynamodbStreamArn`|String|true|" " (empty string)|The DynamoDB stream arn. +`awsCredentialPluginName`|String|false|" " (empty string)|The fully-qualified class name of implementation of {@inject: github:AwsCredentialProviderPlugin:/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java}.

    `awsCredentialProviderPlugin` has the following built-in plugs:

  • `org.apache.pulsar.io.kinesis.AwsDefaultProviderChainPlugin`:
    this plugin uses the default AWS provider chain.
    For more information, see [using the default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

  • `org.apache.pulsar.io.kinesis.STSAssumeRoleProviderPlugin`:
    this plugin takes a configuration via the `awsCredentialPluginParam` that describes a role to assume when running the KCL.
    **JSON configuration example**
    `{"roleArn": "arn...", "roleSessionName": "name"}`

    `awsCredentialPluginName` is a factory class which creates an AWSCredentialsProvider that is used by Kinesis sink.

    If `awsCredentialPluginName` set to empty, the Kinesis sink creates a default AWSCredentialsProvider which accepts json-map of credentials in `awsCredentialPluginParam`.
    • +`awsCredentialPluginParam`|String |false|" " (empty string)|The JSON parameter to initialize `awsCredentialsProviderPlugin`. + +### Example + +Before using the DynamoDB source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "awsEndpoint": "https://some.endpoint.aws", + "awsRegion": "us-east-1", + "awsDynamodbStreamArn": "arn:aws:dynamodb:us-west-2:111122223333:table/TestTable/stream/2015-05-11T21:21:33.291", + "awsCredentialPluginParam": "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}", + "applicationName": "My test application", + "checkpointInterval": "30000", + "backoffTime": "4000", + "numRetries": "3", + "receiveQueueSize": 2000, + "initialPositionInStream": "TRIM_HORIZON", + "startAtTime": "2019-03-05T19:28:58.000Z" + } + + ``` + +* YAML + + ```yaml + + configs: + awsEndpoint: "https://some.endpoint.aws" + awsRegion: "us-east-1" + awsDynamodbStreamArn: "arn:aws:dynamodb:us-west-2:111122223333:table/TestTable/stream/2015-05-11T21:21:33.291" + awsCredentialPluginParam: "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}" + applicationName: "My test application" + checkpointInterval: 30000 + backoffTime: 4000 + numRetries: 3 + receiveQueueSize: 2000 + initialPositionInStream: "TRIM_HORIZON" + startAtTime: "2019-03-05T19:28:58.000Z" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-elasticsearch-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-elasticsearch-sink.md new file mode 100644 index 0000000000000..a2624bdca129d --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-elasticsearch-sink.md @@ -0,0 +1,177 @@ +--- +id: io-elasticsearch-sink +title: ElasticSearch sink connector +sidebar_label: "ElasticSearch sink connector" +original_id: io-elasticsearch-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The ElasticSearch sink connector pulls messages from Pulsar topics and persists the messages to indexes. + +## Configuration + +The configuration of the ElasticSearch sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `elasticSearchUrl` | String| true |" " (empty string)| The URL of elastic search cluster to which the connector connects. | +| `indexName` | String| true |" " (empty string)| The index name to which the connector writes messages. | +| `typeName` | String | false | "_doc" | The type name to which the connector writes messages to.

    The value should be set explicitly to a valid type name other than "_doc" for Elasticsearch version before 6.2, and left to default otherwise. | +| `indexNumberOfShards` | int| false |1| The number of shards of the index. | +| `indexNumberOfReplicas` | int| false |1 | The number of replicas of the index. | +| `username` | String| false |" " (empty string)| The username used by the connector to connect to the elastic search cluster.

    If `username` is set, then `password` should also be provided. | +| `password` | String| false | " " (empty string)|The password used by the connector to connect to the elastic search cluster.

    If `username` is set, then `password` should also be provided. | + +## Example + +Before using the ElasticSearch sink connector, you need to create a configuration file through one of the following methods. + +### Configuration + +#### For Elasticsearch After 6.2 + +* JSON + + ```json + + { + "elasticSearchUrl": "http://localhost:9200", + "indexName": "my_index", + "username": "scooby", + "password": "doobie" + } + + ``` + +* YAML + + ```yaml + + configs: + elasticSearchUrl: "http://localhost:9200" + indexName: "my_index" + username: "scooby" + password: "doobie" + + ``` + +#### For Elasticsearch Before 6.2 + +* JSON + + ```json + + { + "elasticSearchUrl": "http://localhost:9200", + "indexName": "my_index", + "typeName": "doc", + "username": "scooby", + "password": "doobie" + } + + ``` + +* YAML + + ```yaml + + configs: + elasticSearchUrl: "http://localhost:9200" + indexName: "my_index" + typeName: "doc" + username: "scooby" + password: "doobie" + + ``` + +### Usage + +1. Start a single node Elasticsearch cluster. + + ```bash + + $ docker run -p 9200:9200 -p 9300:9300 \ + -e "discovery.type=single-node" \ + docker.elastic.co/elasticsearch/elasticsearch:7.5.1 + + ``` + +2. Start a Pulsar service locally in standalone mode. + + ```bash + + $ bin/pulsar standalone + + ``` + + Make sure the NAR file is available at `connectors/pulsar-io-elastic-search-@pulsar:version@.nar`. + +3. Start the Pulsar Elasticsearch connector in local run mode using one of the following methods. + * Use the **JSON** configuration as shown previously. + + ```bash + + $ bin/pulsar-admin sinks localrun \ + --archive connectors/pulsar-io-elastic-search-@pulsar:version@.nar \ + --tenant public \ + --namespace default \ + --name elasticsearch-test-sink \ + --sink-config '{"elasticSearchUrl":"http://localhost:9200","indexName": "my_index","username": "scooby","password": "doobie"}' \ + --inputs elasticsearch_test + + ``` + + * Use the **YAML** configuration file as shown previously. + + ```bash + + $ bin/pulsar-admin sinks localrun \ + --archive connectors/pulsar-io-elastic-search-@pulsar:version@.nar \ + --tenant public \ + --namespace default \ + --name elasticsearch-test-sink \ + --sink-config-file elasticsearch-sink.yml \ + --inputs elasticsearch_test + + ``` + +4. Publish records to the topic. + + ```bash + + $ bin/pulsar-client produce elasticsearch_test --messages "{\"a\":1}" + + ``` + +5. Check documents in Elasticsearch. + + * refresh the index + + ```bash + + $ curl -s http://localhost:9200/my_index/_refresh + + ``` + + + * search documents + + ```bash + + $ curl -s http://localhost:9200/my_index/_search + + ``` + + You can see the record that published earlier has been successfully written into Elasticsearch. + + ```json + + {"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":"my_index","_type":"_doc","_id":"FSxemm8BLjG_iC0EeTYJ","_score":1.0,"_source":{"a":1}}]}} + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-file-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-file-source.md new file mode 100644 index 0000000000000..4e34dfd9a1f02 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-file-source.md @@ -0,0 +1,163 @@ +--- +id: io-file-source +title: File source connector +sidebar_label: "File source connector" +original_id: io-file-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The File source connector pulls messages from files in directories and persists the messages to Pulsar topics. + +## Configuration + +The configuration of the File source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `inputDirectory` | String|true | No default value|The input directory to pull files. | +| `recurse` | Boolean|false | true | Whether to pull files from subdirectory or not.| +| `keepFile` |Boolean|false | false | If set to true, the file is not deleted after it is processed, which means the file can be picked up continually. | +| `fileFilter` | String|false| [^\\.].* | The file whose name matches the given regular expression is picked up. | +| `pathFilter` | String |false | NULL | If `recurse` is set to true, the subdirectory whose path matches the given regular expression is scanned. | +| `minimumFileAge` | Integer|false | 0 | The minimum age that a file can be processed.

    Any file younger than `minimumFileAge` (according to the last modification date) is ignored. | +| `maximumFileAge` | Long|false |Long.MAX_VALUE | The maximum age that a file can be processed.

    Any file older than `maximumFileAge` (according to last modification date) is ignored. | +| `minimumSize` |Integer| false |1 | The minimum size (in bytes) that a file can be processed. | +| `maximumSize` | Double|false |Double.MAX_VALUE| The maximum size (in bytes) that a file can be processed. | +| `ignoreHiddenFiles` |Boolean| false | true| Whether the hidden files should be ignored or not. | +| `pollingInterval`|Long | false | 10000L | Indicates how long to wait before performing a directory listing. | +| `numWorkers` | Integer | false | 1 | The number of worker threads that process files.

    This allows you to process a larger number of files concurrently.

    However, setting this to a value greater than 1 makes the data from multiple files mixed in the target topic. | + +### Example + +Before using the File source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "inputDirectory": "/Users/david", + "recurse": true, + "keepFile": true, + "fileFilter": "[^\\.].*", + "pathFilter": "*", + "minimumFileAge": 0, + "maximumFileAge": 9999999999, + "minimumSize": 1, + "maximumSize": 5000000, + "ignoreHiddenFiles": true, + "pollingInterval": 5000, + "numWorkers": 1 + } + + ``` + +* YAML + + ```yaml + + configs: + inputDirectory: "/Users/david" + recurse: true + keepFile: true + fileFilter: "[^\\.].*" + pathFilter: "*" + minimumFileAge: 0 + maximumFileAge: 9999999999 + minimumSize: 1 + maximumSize: 5000000 + ignoreHiddenFiles: true + pollingInterval: 5000 + numWorkers: 1 + + ``` + +## Usage + +Here is an example of using the File source connecter. + +1. Pull a Pulsar image. + + ```bash + + $ docker pull apachepulsar/pulsar:{version} + + ``` + +2. Start Pulsar standalone. + + ```bash + + $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:{version} bin/pulsar standalone + + ``` + +3. Create a configuration file _file-connector.yaml_. + + ```yaml + + configs: + inputDirectory: "/opt" + + ``` + +4. Copy the configuration file _file-connector.yaml_ to the container. + + ```bash + + $ docker cp connectors/file-connector.yaml pulsar-standalone:/pulsar/ + + ``` + +5. Download the File source connector. + + ```bash + + $ curl -O https://mirrors.tuna.tsinghua.edu.cn/apache/pulsar/pulsar-{version}/connectors/pulsar-io-file-{version}.nar + + ``` + +6. Start the File source connector. + + ```bash + + $ docker exec -it pulsar-standalone /bin/bash + + $ ./bin/pulsar-admin sources localrun \ + --archive /pulsar/pulsar-io-file-{version}.nar \ + --name file-test \ + --destination-topic-name pulsar-file-test \ + --source-config-file /pulsar/file-connector.yaml + + ``` + +7. Start a consumer. + + ```bash + + ./bin/pulsar-client consume -s file-test -n 0 pulsar-file-test + + ``` + +8. Write the message to the file _test.txt_. + + ```bash + + echo "hello world!" > /opt/test.txt + + ``` + + The following information appears on the consumer terminal window. + + ```bash + + ----- got message ----- + hello world! + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-flume-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-flume-sink.md new file mode 100644 index 0000000000000..ded05d80726f1 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-flume-sink.md @@ -0,0 +1,60 @@ +--- +id: io-flume-sink +title: Flume sink connector +sidebar_label: "Flume sink connector" +original_id: io-flume-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Flume sink connector pulls messages from Pulsar topics to logs. + +## Configuration + +The configuration of the Flume sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +`name`|String|true|"" (empty string)|The name of the agent. +`confFile`|String|true|"" (empty string)|The configuration file. +`noReloadConf`|Boolean|false|false|Whether to reload configuration file if changed. +`zkConnString`|String|true|"" (empty string)|The ZooKeeper connection. +`zkBasePath`|String|true|"" (empty string)|The base path in ZooKeeper for agent configuration. + +### Example + +Before using the Flume sink connector, you need to create a configuration file through one of the following methods. + +> For more information about the `sink.conf` in the example below, see [here](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/sink.conf). + +* JSON + + ```json + + { + "name": "a1", + "confFile": "sink.conf", + "noReloadConf": "false", + "zkConnString": "", + "zkBasePath": "" + } + + ``` + +* YAML + + ```yaml + + configs: + name: a1 + confFile: sink.conf + noReloadConf: false + zkConnString: "" + zkBasePath: "" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-flume-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-flume-source.md new file mode 100644 index 0000000000000..42f35e7ea499d --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-flume-source.md @@ -0,0 +1,60 @@ +--- +id: io-flume-source +title: Flume source connector +sidebar_label: "Flume source connector" +original_id: io-flume-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Flume source connector pulls messages from logs to Pulsar topics. + +## Configuration + +The configuration of the Flume source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +`name`|String|true|"" (empty string)|The name of the agent. +`confFile`|String|true|"" (empty string)|The configuration file. +`noReloadConf`|Boolean|false|false|Whether to reload configuration file if changed. +`zkConnString`|String|true|"" (empty string)|The ZooKeeper connection. +`zkBasePath`|String|true|"" (empty string)|The base path in ZooKeeper for agent configuration. + +### Example + +Before using the Flume source connector, you need to create a configuration file through one of the following methods. + +> For more information about the `source.conf` in the example below, see [here](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/source.conf). + +* JSON + + ```json + + { + "name": "a1", + "confFile": "source.conf", + "noReloadConf": "false", + "zkConnString": "", + "zkBasePath": "" + } + + ``` + +* YAML + + ```yaml + + configs: + name: a1 + confFile: source.conf + noReloadConf: false + zkConnString: "" + zkBasePath: "" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-hbase-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-hbase-sink.md new file mode 100644 index 0000000000000..8d39551a05ddb --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-hbase-sink.md @@ -0,0 +1,70 @@ +--- +id: io-hbase-sink +title: HBase sink connector +sidebar_label: "HBase sink connector" +original_id: io-hbase-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The HBase sink connector pulls the messages from Pulsar topics +and persists the messages to HBase tables + +## Configuration + +The configuration of the HBase sink connector has the following properties. + +### Property + +| Name | Type|Default | Required | Description | +|------|---------|----------|-------------|--- +| `hbaseConfigResources` | String|None | false | HBase system configuration `hbase-site.xml` file. | +| `zookeeperQuorum` | String|None | true | HBase system configuration about `hbase.zookeeper.quorum` value. | +| `zookeeperClientPort` | String|2181 | false | HBase system configuration about `hbase.zookeeper.property.clientPort` value. | +| `zookeeperZnodeParent` | String|/hbase | false | HBase system configuration about `zookeeper.znode.parent` value. | +| `tableName` | None |String | true | HBase table, the value is `namespace:tableName`. | +| `rowKeyName` | String|None | true | HBase table rowkey name. | +| `familyName` | String|None | true | HBase table column family name. | +| `qualifierNames` |String| None | true | HBase table column qualifier names. | +| `batchTimeMs` | Long|1000l| false | HBase table operation timeout in milliseconds. | +| `batchSize` | int|200| false | Batch size of updates made to the HBase table. | + +### Example + +Before using the HBase sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "hbaseConfigResources": "hbase-site.xml", + "zookeeperQuorum": "localhost", + "zookeeperClientPort": "2181", + "zookeeperZnodeParent": "/hbase", + "tableName": "pulsar_hbase", + "rowKeyName": "rowKey", + "familyName": "info", + "qualifierNames": [ 'name', 'address', 'age'] + } + + ``` + +* YAML + + ```yaml + + configs: + hbaseConfigResources: "hbase-site.xml" + zookeeperQuorum: "localhost" + zookeeperClientPort: "2181" + zookeeperZnodeParent: "/hbase" + tableName: "pulsar_hbase" + rowKeyName: "rowKey" + familyName: "info" + qualifierNames: [ 'name', 'address', 'age'] + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-hdfs2-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-hdfs2-sink.md new file mode 100644 index 0000000000000..10369997588d6 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-hdfs2-sink.md @@ -0,0 +1,68 @@ +--- +id: io-hdfs2-sink +title: HDFS2 sink connector +sidebar_label: "HDFS2 sink connector" +original_id: io-hdfs2-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The HDFS2 sink connector pulls the messages from Pulsar topics +and persists the messages to HDFS files. + +## Configuration + +The configuration of the HDFS2 sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `hdfsConfigResources` | String|true| None | A file or a comma-separated list containing the Hadoop file system configuration.

    **Example**
    'core-site.xml'
    'hdfs-site.xml' | +| `directory` | String | true | None|The HDFS directory where files read from or written to. | +| `encoding` | String |false |None |The character encoding for the files.

    **Example**
    UTF-8
    ASCII | +| `compression` | Compression |false |None |The compression code used to compress or de-compress the files on HDFS.

    Below are the available options:
  • BZIP2
  • DEFLATE
  • GZIP
  • LZ4
  • SNAPPY
    • | +| `kerberosUserPrincipal` |String| false| None|The principal account of Kerberos user used for authentication. | +| `keytab` | String|false|None| The full pathname of the Kerberos keytab file used for authentication. | +| `filenamePrefix` |String| true, if `compression` is set to `None`. | None |The prefix of the files created inside the HDFS directory.

    **Example**
    The value of topicA result in files named topicA-. | +| `fileExtension` | String| true | None | The extension added to the files written to HDFS.

    **Example**
    '.txt'
    '.seq' | +| `separator` | char|false |None |The character used to separate records in a text file.

    If no value is provided, the contents from all records are concatenated together in one continuous byte array. | +| `syncInterval` | long| false |0| The interval between calls to flush data to HDFS disk in milliseconds. | +| `maxPendingRecords` |int| false|Integer.MAX_VALUE | The maximum number of records that hold in memory before acking.

    Setting this property to 1 makes every record send to disk before the record is acked.

    Setting this property to a higher value allows buffering records before flushing them to disk. +| `subdirectoryPattern` | String | false | None | A subdirectory associated with the created time of the sink.
    The pattern is the formatted pattern of `directory`'s subdirectory.

    See [DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) for pattern's syntax. | + +### Example + +Before using the HDFS2 sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "hdfsConfigResources": "core-site.xml", + "directory": "/foo/bar", + "filenamePrefix": "prefix", + "fileExtension": ".log", + "compression": "SNAPPY", + "subdirectoryPattern": "yyyy-MM-dd" + } + + ``` + +* YAML + + ```yaml + + configs: + hdfsConfigResources: "core-site.xml" + directory: "/foo/bar" + filenamePrefix: "prefix" + fileExtension: ".log" + compression: "SNAPPY" + subdirectoryPattern: "yyyy-MM-dd" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-hdfs3-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-hdfs3-sink.md new file mode 100644 index 0000000000000..6bd77ee58d277 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-hdfs3-sink.md @@ -0,0 +1,63 @@ +--- +id: io-hdfs3-sink +title: HDFS3 sink connector +sidebar_label: "HDFS3 sink connector" +original_id: io-hdfs3-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The HDFS3 sink connector pulls the messages from Pulsar topics +and persists the messages to HDFS files. + +## Configuration + +The configuration of the HDFS3 sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `hdfsConfigResources` | String|true| None | A file or a comma-separated list containing the Hadoop file system configuration.

    **Example**
    'core-site.xml'
    'hdfs-site.xml' | +| `directory` | String | true | None|The HDFS directory where files read from or written to. | +| `encoding` | String |false |None |The character encoding for the files.

    **Example**
    UTF-8
    ASCII | +| `compression` | Compression |false |None |The compression code used to compress or de-compress the files on HDFS.

    Below are the available options:
  • BZIP2
  • DEFLATE
  • GZIP
  • LZ4
  • SNAPPY
    • | +| `kerberosUserPrincipal` |String| false| None|The principal account of Kerberos user used for authentication. | +| `keytab` | String|false|None| The full pathname of the Kerberos keytab file used for authentication. | +| `filenamePrefix` |String| false |None |The prefix of the files created inside the HDFS directory.

    **Example**
    The value of topicA result in files named topicA-. | +| `fileExtension` | String| false | None| The extension added to the files written to HDFS.

    **Example**
    '.txt'
    '.seq' | +| `separator` | char|false |None |The character used to separate records in a text file.

    If no value is provided, the contents from all records are concatenated together in one continuous byte array. | +| `syncInterval` | long| false |0| The interval between calls to flush data to HDFS disk in milliseconds. | +| `maxPendingRecords` |int| false|Integer.MAX_VALUE | The maximum number of records that hold in memory before acking.

    Setting this property to 1 makes every record send to disk before the record is acked.

    Setting this property to a higher value allows buffering records before flushing them to disk. + +### Example + +Before using the HDFS3 sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "hdfsConfigResources": "core-site.xml", + "directory": "/foo/bar", + "filenamePrefix": "prefix", + "compression": "SNAPPY" + } + + ``` + +* YAML + + ```yaml + + configs: + hdfsConfigResources: "core-site.xml" + directory: "/foo/bar" + filenamePrefix: "prefix" + compression: "SNAPPY" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-influxdb-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-influxdb-sink.md new file mode 100644 index 0000000000000..4023423964e34 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-influxdb-sink.md @@ -0,0 +1,123 @@ +--- +id: io-influxdb-sink +title: InfluxDB sink connector +sidebar_label: "InfluxDB sink connector" +original_id: io-influxdb-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The InfluxDB sink connector pulls messages from Pulsar topics +and persists the messages to InfluxDB. + +The InfluxDB sink provides different configurations for InfluxDBv1 and v2 respectively. + +## Configuration + +The configuration of the InfluxDB sink connector has the following properties. + +### Property +#### InfluxDBv2 +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `influxdbUrl` |String| true|" " (empty string) | The URL of the InfluxDB instance. | +| `token` | String|true| " " (empty string) |The authentication token used to authenticate to InfluxDB. | +| `organization` | String| true|" " (empty string) | The InfluxDB organization to write to. | +| `bucket` |String| true | " " (empty string)| The InfluxDB bucket to write to. | +| `precision` | String|false| ns | The timestamp precision for writing data to InfluxDB.

    Below are the available options:
  • ns
  • us
  • ms
  • s
    • | +| `logLevel` | String|false| NONE|The log level for InfluxDB request and response.

    Below are the available options:
  • NONE
  • BASIC
  • HEADERS
  • FULL
    • | +| `gzipEnable` | boolean|false | false | Whether to enable gzip or not. | +| `batchTimeMs` |long|false| 1000L | The InfluxDB operation time in milliseconds. | +| `batchSize` | int|false|200| The batch size of writing to InfluxDB. | + +#### InfluxDBv1 +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `influxdbUrl` |String| true|" " (empty string) | The URL of the InfluxDB instance. | +| `username` | String|false| " " (empty string) |The username used to authenticate to InfluxDB. | +| `password` | String| false|" " (empty string) | The password used to authenticate to InfluxDB. | +| `database` |String| true | " " (empty string)| The InfluxDB to which write messages. | +| `consistencyLevel` | String|false|ONE | The consistency level for writing data to InfluxDB.

    Below are the available options:
  • ALL
  • ANY
  • ONE
  • QUORUM
    • | +| `logLevel` | String|false| NONE|The log level for InfluxDB request and response.

    Below are the available options:
  • NONE
  • BASIC
  • HEADERS
  • FULL
    • | +| `retentionPolicy` | String|false| autogen| The retention policy for InfluxDB. | +| `gzipEnable` | boolean|false | false | Whether to enable gzip or not. | +| `batchTimeMs` |long|false| 1000L | The InfluxDB operation time in milliseconds. | +| `batchSize` | int|false|200| The batch size of writing to InfluxDB. | + +### Example +Before using the InfluxDB sink connector, you need to create a configuration file through one of the following methods. +#### InfluxDBv2 +* JSON + + ```json + + { + "influxdbUrl": "http://localhost:9999", + "organization": "example-org", + "bucket": "example-bucket", + "token": "xxxx", + "precision": "ns", + "logLevel": "NONE", + "gzipEnable": false, + "batchTimeMs": 1000, + "batchSize": 100 + } + + ``` + + +* YAML + + ```yaml + + configs: + influxdbUrl: "http://localhost:9999" + organization: "example-org" + bucket: "example-bucket" + token: "xxxx" + precision: "ns" + logLevel: "NONE" + gzipEnable: false + batchTimeMs: 1000 + batchSize: 100 + + ``` + + +#### InfluxDBv1 + +* JSON + + ```json + + { + "influxdbUrl": "http://localhost:8086", + "database": "test_db", + "consistencyLevel": "ONE", + "logLevel": "NONE", + "retentionPolicy": "autogen", + "gzipEnable": false, + "batchTimeMs": 1000, + "batchSize": 100 + } + + ``` + +* YAML + + ```yaml + + configs: + influxdbUrl: "http://localhost:8086" + database: "test_db" + consistencyLevel: "ONE" + logLevel: "NONE" + retentionPolicy: "autogen" + gzipEnable: false + batchTimeMs: 1000 + batchSize: 100 + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-jdbc-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-jdbc-sink.md new file mode 100644 index 0000000000000..e992e724e534d --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-jdbc-sink.md @@ -0,0 +1,161 @@ +--- +id: io-jdbc-sink +title: JDBC sink connector +sidebar_label: "JDBC sink connector" +original_id: io-jdbc-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The JDBC sink connectors allow pulling messages from Pulsar topics +and persists the messages to ClickHouse, MariaDB, PostgreSQL, and SQLite. + +> Currently, INSERT, DELETE and UPDATE operations are supported. + +## Configuration + +The configuration of all JDBC sink connectors has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `userName` | String|false | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.

    **Note: `userName` is case-sensitive.**| +| `password` | String|false | " " (empty string)| The password used to connect to the database specified by `jdbcUrl`.

    **Note: `password` is case-sensitive.**| +| `jdbcUrl` | String|true | " " (empty string) | The JDBC URL of the database to which the connector connects. | +| `tableName` | String|true | " " (empty string) | The name of the table to which the connector writes. | +| `nonKey` | String|false | " " (empty string) | A comma-separated list contains the fields used in updating events. | +| `key` | String|false | " " (empty string) | A comma-separated list contains the fields used in `where` condition of updating and deleting events. | +| `timeoutMs` | int| false|500 | The JDBC operation timeout in milliseconds. | +| `batchSize` | int|false | 200 | The batch size of updates made to the database. | + +### Example for ClickHouse + +* JSON + + ```json + + { + "userName": "clickhouse", + "password": "password", + "jdbcUrl": "jdbc:clickhouse://localhost:8123/pulsar_clickhouse_jdbc_sink", + "tableName": "pulsar_clickhouse_jdbc_sink" + } + + ``` + +* YAML + + ```yaml + + tenant: "public" + namespace: "default" + name: "jdbc-clickhouse-sink" + topicName: "persistent://public/default/jdbc-clickhouse-topic" + sinkType: "jdbc-clickhouse" + configs: + userName: "clickhouse" + password: "password" + jdbcUrl: "jdbc:clickhouse://localhost:8123/pulsar_clickhouse_jdbc_sink" + tableName: "pulsar_clickhouse_jdbc_sink" + + ``` + +### Example for MariaDB + +* JSON + + ```json + + { + "userName": "mariadb", + "password": "password", + "jdbcUrl": "jdbc:mariadb://localhost:3306/pulsar_mariadb_jdbc_sink", + "tableName": "pulsar_mariadb_jdbc_sink" + } + + ``` + +* YAML + + ```yaml + + tenant: "public" + namespace: "default" + name: "jdbc-mariadb-sink" + topicName: "persistent://public/default/jdbc-mariadb-topic" + sinkType: "jdbc-mariadb" + configs: + userName: "mariadb" + password: "password" + jdbcUrl: "jdbc:mariadb://localhost:3306/pulsar_mariadb_jdbc_sink" + tableName: "pulsar_mariadb_jdbc_sink" + + ``` + +### Example for PostgreSQL + +Before using the JDBC PostgreSQL sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "userName": "postgres", + "password": "password", + "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", + "tableName": "pulsar_postgres_jdbc_sink" + } + + ``` + +* YAML + + ```yaml + + tenant: "public" + namespace: "default" + name: "jdbc-postgres-sink" + topicName: "persistent://public/default/jdbc-postgres-topic" + sinkType: "jdbc-postgres" + configs: + userName: "postgres" + password: "password" + jdbcUrl: "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink" + tableName: "pulsar_postgres_jdbc_sink" + + ``` + +For more information on **how to use this JDBC sink connector**, see [connect Pulsar to PostgreSQL](io-quickstart.md#connect-pulsar-to-postgresql). + +### Example for SQLite + +* JSON + + ```json + + { + "jdbcUrl": "jdbc:sqlite:db.sqlite", + "tableName": "pulsar_sqlite_jdbc_sink" + } + + ``` + +* YAML + + ```yaml + + tenant: "public" + namespace: "default" + name: "jdbc-sqlite-sink" + topicName: "persistent://public/default/jdbc-sqlite-topic" + sinkType: "jdbc-sqlite" + configs: + jdbcUrl: "jdbc:sqlite:db.sqlite" + tableName: "pulsar_sqlite_jdbc_sink" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-kafka-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-kafka-sink.md new file mode 100644 index 0000000000000..743f78514e068 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-kafka-sink.md @@ -0,0 +1,76 @@ +--- +id: io-kafka-sink +title: Kafka sink connector +sidebar_label: "Kafka sink connector" +original_id: io-kafka-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Kafka sink connector pulls messages from Pulsar topics and persists the messages +to Kafka topics. + +This guide explains how to configure and use the Kafka sink connector. + +## Configuration + +The configuration of the Kafka sink connector has the following parameters. + +### Property + +| Name | Type| Required | Default | Description +|------|----------|---------|-------------|-------------| +| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. | +|`acks`|String|true|" " (empty string) |The number of acknowledgments that the producer requires the leader to receive before a request completes.
    This controls the durability of the sent records. +|`batchsize`|long|false|16384L|The batch size that a Kafka producer attempts to batch records together before sending them to brokers. +|`maxRequestSize`|long|false|1048576L|The maximum size of a Kafka request in bytes. +|`topic`|String|true|" " (empty string) |The Kafka topic which receives messages from Pulsar. +| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringSerializer | The serializer class for Kafka producers to serialize keys. +| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArraySerializer | The serializer class for Kafka producers to serialize values.

    The serializer is set by a specific implementation of [`KafkaAbstractSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java). +|`producerConfigProperties`|Map|false|" " (empty string)|The producer configuration properties to be passed to producers.

    **Note: other properties specified in the connector configuration file take precedence over this configuration**. + + +### Example + +Before using the Kafka sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "bootstrapServers": "localhost:6667", + "topic": "test", + "acks": "1", + "batchSize": "16384", + "maxRequestSize": "1048576", + "producerConfigProperties": + { + "client.id": "test-pulsar-producer", + "security.protocol": "SASL_PLAINTEXT", + "sasl.mechanism": "GSSAPI", + "sasl.kerberos.service.name": "kafka", + "acks": "all" + } + } + +* YAML + + ``` + +yaml + configs: + bootstrapServers: "localhost:6667" + topic: "test" + acks: "1" + batchSize: "16384" + maxRequestSize: "1048576" + producerConfigProperties: + client.id: "test-pulsar-producer" + security.protocol: "SASL_PLAINTEXT" + sasl.mechanism: "GSSAPI" + sasl.kerberos.service.name: "kafka" + acks: "all" + ``` diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-kafka-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-kafka-source.md new file mode 100644 index 0000000000000..e353aea21c4b7 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-kafka-source.md @@ -0,0 +1,201 @@ +--- +id: io-kafka-source +title: Kafka source connector +sidebar_label: "Kafka source connector" +original_id: io-kafka-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Kafka source connector pulls messages from Kafka topics and persists the messages +to Pulsar topics. + +This guide explains how to configure and use the Kafka source connector. + +## Configuration + +The configuration of the Kafka source connector has the following properties. + +### Property + +| Name | Type| Required | Default | Description +|------|----------|---------|-------------|-------------| +| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. | +| `groupId` |String| true | " " (empty string) | A unique string that identifies the group of consumer processes to which this consumer belongs. | +| `fetchMinBytes` | long|false | 1 | The minimum byte expected for each fetch response. | +| `autoCommitEnabled` | boolean |false | true | If set to true, the consumer's offset is periodically committed in the background.

    This committed offset is used when the process fails as the position from which a new consumer begins. | +| `autoCommitIntervalMs` | long|false | 5000 | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true. | +| `heartbeatIntervalMs` | long| false | 3000 | The interval between heartbeats to the consumer when using Kafka's group management facilities.

    **Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.| +| `sessionTimeoutMs` | long|false | 30000 | The timeout used to detect consumer failures when using Kafka's group management facility. | +| `topic` | String|true | " " (empty string)| The Kafka topic which sends messages to Pulsar. | +| `consumerConfigProperties` | Map| false | " " (empty string) | The consumer configuration properties to be passed to consumers.

    **Note: other properties specified in the connector configuration file take precedence over this configuration**. | +| `keyDeserializationClass` | String|false | org.apache.kafka.common.serialization.StringDeserializer | The deserializer class for Kafka consumers to deserialize keys.
    The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java). +| `valueDeserializationClass` | String|false | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values. + + +### Example + +Before using the Kafka source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "bootstrapServers": "pulsar-kafka:9092", + "groupId": "test-pulsar-io", + "topic": "my-topic", + "sessionTimeoutMs": "10000", + "autoCommitEnabled": false + } + + ``` + +* YAML + + ```yaml + + configs: + bootstrapServers: "pulsar-kafka:9092" + groupId: "test-pulsar-io" + topic: "my-topic" + sessionTimeoutMs: "10000" + autoCommitEnabled: false + + ``` + +## Usage + +Here is an example of using the Kafka source connecter with the configuration file as shown previously. + +1. Download a Kafka client and a Kafka connector. + + ```bash + + $ wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar + + $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/connectors/pulsar-io-kafka-2.4.0.nar + + ``` + +2. Create a network. + + ```bash + + $ docker network create kafka-pulsar + + ``` + +3. Pull a ZooKeeper image and start ZooKeeper. + + ```bash + + $ docker pull wurstmeister/zookeeper + + $ docker run -d -it -p 2181:2181 --name pulsar-kafka-zookeeper --network kafka-pulsar wurstmeister/zookeeper + + ``` + +4. Pull a Kafka image and start Kafka. + + ```bash + + $ docker pull wurstmeister/kafka:2.11-1.0.2 + + $ docker run -d -it --network kafka-pulsar -p 6667:6667 -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME=pulsar-kafka -e KAFKA_ZOOKEEPER_CONNECT=pulsar-kafka-zookeeper:2181 --name pulsar-kafka wurstmeister/kafka:2.11-1.0.2 + + ``` + +5. Pull a Pulsar image and start Pulsar standalone. + + ```bash + + $ docker pull apachepulsar/pulsar:2.4.0 + + $ docker run -d -it --network kafka-pulsar -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-kafka-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone + + ``` + +6. Create a producer file _kafka-producer.py_. + + ```python + + from kafka import KafkaProducer + producer = KafkaProducer(bootstrap_servers='pulsar-kafka:9092') + future = producer.send('my-topic', b'hello world') + future.get() + + ``` + +7. Create a consumer file _pulsar-client.py_. + + ```python + + import pulsar + + client = pulsar.Client('pulsar://localhost:6650') + consumer = client.subscribe('my-topic', subscription_name='my-aa') + + while True: + msg = consumer.receive() + print msg + print dir(msg) + print("Received message: '%s'" % msg.data()) + consumer.acknowledge(msg) + + client.close() + + ``` + +8. Copy the following files to Pulsar. + + ```bash + + $ docker cp pulsar-io-kafka-2.4.0.nar pulsar-kafka-standalone:/pulsar + $ docker cp kafkaSourceConfig.yaml pulsar-kafka-standalone:/pulsar/conf + $ docker cp kafka-clients-0.10.2.1.jar pulsar-kafka-standalone:/pulsar/lib + $ docker cp pulsar-client.py pulsar-kafka-standalone:/pulsar/ + $ docker cp kafka-producer.py pulsar-kafka-standalone:/pulsar/ + + ``` + +9. Open a new terminal window and start the Kafka source connector in local run mode. + + ```bash + + $ docker exec -it pulsar-kafka-standalone /bin/bash + + $ ./bin/pulsar-admin source localrun \ + --archive ./pulsar-io-kafka-2.4.0.nar \ + --classname org.apache.pulsar.io.kafka.KafkaBytesSource \ + --tenant public \ + --namespace default \ + --name kafka \ + --destination-topic-name my-topic \ + --source-config-file ./conf/kafkaSourceConfig.yaml \ + --parallelism 1 + + ``` + +10. Open a new terminal window and run the consumer. + + ```bash + + $ docker exec -it pulsar-kafka-standalone /bin/bash + + $ pip install kafka-python + + $ python3 kafka-producer.py + + ``` + + The following information appears on the consumer terminal window. + + ```bash + + Received message: 'hello world' + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-sink.md new file mode 100644 index 0000000000000..483c861db284d --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-sink.md @@ -0,0 +1,84 @@ +--- +id: io-kinesis-sink +title: Kinesis sink connector +sidebar_label: "Kinesis sink connector" +original_id: io-kinesis-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Kinesis sink connector pulls data from Pulsar and persists data into Amazon Kinesis. + +## Configuration + +The configuration of the Kinesis sink connector has the following property. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +`messageFormat`|MessageFormat|true|ONLY_RAW_PAYLOAD|Message format in which Kinesis sink converts Pulsar messages and publishes to Kinesis streams.

    Below are the available options:

  • `ONLY_RAW_PAYLOAD`: Kinesis sink directly publishes Pulsar message payload as a message into the configured Kinesis stream.

  • `FULL_MESSAGE_IN_JSON`: Kinesis sink creates a JSON payload with Pulsar message payload, properties and encryptionCtx, and publishes JSON payload into the configured Kinesis stream.

  • `FULL_MESSAGE_IN_FB`: Kinesis sink creates a flatbuffer serialized payload with Pulsar message payload, properties and encryptionCtx, and publishes flatbuffer payload into the configured Kinesis stream.
    • +`retainOrdering`|boolean|false|false|Whether Pulsar connectors to retain ordering when moving messages from Pulsar to Kinesis or not. +`awsEndpoint`|String|false|" " (empty string)|The Kinesis end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`awsRegion`|String|false|" " (empty string)|The AWS region.

    **Example**
    us-west-1, us-west-2 +`awsKinesisStreamName`|String|true|" " (empty string)|The Kinesis stream name. +`awsCredentialPluginName`|String|false|" " (empty string)|The fully-qualified class name of implementation of {@inject: github:AwsCredentialProviderPlugin:/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java}.

    It is a factory class which creates an AWSCredentialsProvider that is used by Kinesis sink.

    If it is empty, the Kinesis sink creates a default AWSCredentialsProvider which accepts json-map of credentials in `awsCredentialPluginParam`. +`awsCredentialPluginParam`|String |false|" " (empty string)|The JSON parameter to initialize `awsCredentialsProviderPlugin`. + +### Built-in plugins + +The following are built-in `AwsCredentialProviderPlugin` plugins: + +* `org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin` + + This plugin takes no configuration, it uses the default AWS provider chain. + + For more information, see [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default). + +* `org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin` + + This plugin takes a configuration (via the `awsCredentialPluginParam`) that describes a role to assume when running the KCL. + + This configuration takes the form of a small json document like: + + ```json + + {"roleArn": "arn...", "roleSessionName": "name"} + + ``` + +### Example + +Before using the Kinesis sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "awsEndpoint": "some.endpoint.aws", + "awsRegion": "us-east-1", + "awsKinesisStreamName": "my-stream", + "awsCredentialPluginParam": "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}", + "messageFormat": "ONLY_RAW_PAYLOAD", + "retainOrdering": "true" + } + + ``` + +* YAML + + ```yaml + + configs: + awsEndpoint: "some.endpoint.aws" + awsRegion: "us-east-1" + awsKinesisStreamName: "my-stream" + awsCredentialPluginParam: "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}" + messageFormat: "ONLY_RAW_PAYLOAD" + retainOrdering: "true" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-source.md new file mode 100644 index 0000000000000..3e593a31161ad --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-kinesis-source.md @@ -0,0 +1,85 @@ +--- +id: io-kinesis-source +title: Kinesis source connector +sidebar_label: "Kinesis source connector" +original_id: io-kinesis-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Kinesis source connector pulls data from Amazon Kinesis and persists data into Pulsar. + +This connector uses the [Kinesis Consumer Library](https://github.com/awslabs/amazon-kinesis-client) (KCL) to do the actual consuming of messages. The KCL uses DynamoDB to track state for consumers. + +> Note: currently, the Kinesis source connector only supports raw messages. If you use KMS encrypted messages, the encrypted messages are sent to downstream. This connector will support decrypting messages in the future release. + + +## Configuration + +The configuration of the Kinesis source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +`initialPositionInStream`|InitialPositionInStream|false|LATEST|The position where the connector starts from.

    Below are the available options:

  • `AT_TIMESTAMP`: start from the record at or after the specified timestamp.

  • `LATEST`: start after the most recent data record.

  • `TRIM_HORIZON`: start from the oldest available data record.
    • +`startAtTime`|Date|false|" " (empty string)|If set to `AT_TIMESTAMP`, it specifies the point in time to start consumption. +`applicationName`|String|false|Pulsar IO connector|The name of the Amazon Kinesis application.

    By default, the application name is included in the user agent string used to make AWS requests. This can assist with troubleshooting, for example, distinguish requests made by separate connector instances. +`checkpointInterval`|long|false|60000|The frequency of the Kinesis stream checkpoint in milliseconds. +`backoffTime`|long|false|3000|The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds. +`numRetries`|int|false|3|The number of re-attempts when the connector encounters an exception while trying to set a checkpoint. +`receiveQueueSize`|int|false|1000|The maximum number of AWS records that can be buffered inside the connector.

    Once the `receiveQueueSize` is reached, the connector does not consume any messages from Kinesis until some messages in the queue are successfully consumed. +`dynamoEndpoint`|String|false|" " (empty string)|The Dynamo end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`cloudwatchEndpoint`|String|false|" " (empty string)|The Cloudwatch end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`useEnhancedFanOut`|boolean|false|true|If set to true, it uses Kinesis enhanced fan-out.

    If set to false, it uses polling. +`awsEndpoint`|String|false|" " (empty string)|The Kinesis end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html). +`awsRegion`|String|false|" " (empty string)|The AWS region.

    **Example**
    us-west-1, us-west-2 +`awsKinesisStreamName`|String|true|" " (empty string)|The Kinesis stream name. +`awsCredentialPluginName`|String|false|" " (empty string)|The fully-qualified class name of implementation of {@inject: github:AwsCredentialProviderPlugin:/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java}.

    `awsCredentialProviderPlugin` has the following built-in plugs:

  • `org.apache.pulsar.io.kinesis.AwsDefaultProviderChainPlugin`:
    this plugin uses the default AWS provider chain.
    For more information, see [using the default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

  • `org.apache.pulsar.io.kinesis.STSAssumeRoleProviderPlugin`:
    this plugin takes a configuration via the `awsCredentialPluginParam` that describes a role to assume when running the KCL.
    **JSON configuration example**
    `{"roleArn": "arn...", "roleSessionName": "name"}`

    `awsCredentialPluginName` is a factory class which creates an AWSCredentialsProvider that is used by Kinesis sink.

    If `awsCredentialPluginName` set to empty, the Kinesis sink creates a default AWSCredentialsProvider which accepts json-map of credentials in `awsCredentialPluginParam`.
    • +`awsCredentialPluginParam`|String |false|" " (empty string)|The JSON parameter to initialize `awsCredentialsProviderPlugin`. + +### Example + +Before using the Kinesis source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "awsEndpoint": "https://some.endpoint.aws", + "awsRegion": "us-east-1", + "awsKinesisStreamName": "my-stream", + "awsCredentialPluginParam": "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}", + "applicationName": "My test application", + "checkpointInterval": "30000", + "backoffTime": "4000", + "numRetries": "3", + "receiveQueueSize": 2000, + "initialPositionInStream": "TRIM_HORIZON", + "startAtTime": "2019-03-05T19:28:58.000Z" + } + + ``` + +* YAML + + ```yaml + + configs: + awsEndpoint: "https://some.endpoint.aws" + awsRegion: "us-east-1" + awsKinesisStreamName: "my-stream" + awsCredentialPluginParam: "{\"accessKey\":\"myKey\",\"secretKey\":\"my-Secret\"}" + applicationName: "My test application" + checkpointInterval: 30000 + backoffTime: 4000 + numRetries: 3 + receiveQueueSize: 2000 + initialPositionInStream: "TRIM_HORIZON" + startAtTime: "2019-03-05T19:28:58.000Z" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-mongo-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-mongo-sink.md new file mode 100644 index 0000000000000..b370464c18f4b --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-mongo-sink.md @@ -0,0 +1,61 @@ +--- +id: io-mongo-sink +title: MongoDB sink connector +sidebar_label: "MongoDB sink connector" +original_id: io-mongo-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The MongoDB sink connector pulls messages from Pulsar topics +and persists the messages to collections. + +## Configuration + +The configuration of the MongoDB sink connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `mongoUri` | String| true| " " (empty string) | The MongoDB URI to which the connector connects.

    For more information, see [connection string URI format](https://docs.mongodb.com/manual/reference/connection-string/). | +| `database` | String| true| " " (empty string)| The database name to which the collection belongs. | +| `collection` | String| true| " " (empty string)| The collection name to which the connector writes messages. | +| `batchSize` | int|false|100 | The batch size of writing messages to collections. | +| `batchTimeMs` |long|false|1000| The batch operation interval in milliseconds. | + + +### Example + +Before using the Mongo sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "mongoUri": "mongodb://localhost:27017", + "database": "pulsar", + "collection": "messages", + "batchSize": "2", + "batchTimeMs": "500" + } + + ``` + +* YAML + + ```yaml + + { + mongoUri: "mongodb://localhost:27017" + database: "pulsar" + collection: "messages" + batchSize: 2 + batchTimeMs: 500 + } + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-netty-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-netty-source.md new file mode 100644 index 0000000000000..d41265fa4fa28 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-netty-source.md @@ -0,0 +1,245 @@ +--- +id: io-netty-source +title: Netty source connector +sidebar_label: "Netty source connector" +original_id: io-netty-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Netty source connector opens a port that accepts incoming data via the configured network protocol +and publish it to user-defined Pulsar topics. + +This connector can be used in a containerized (for example, k8s) deployment. Otherwise, if the connector is running in process or thread mode, the instance may be conflicting on listening to ports. + +## Configuration + +The configuration of the Netty source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `type` |String| true |tcp | The network protocol over which data is transmitted to netty.

    Below are the available options:
  • tcp
  • http
  • udp
    • | +| `host` | String|true | 127.0.0.1 | The host name or address on which the source instance listen. | +| `port` | int|true | 10999 | The port on which the source instance listen. | +| `numberOfThreads` |int| true |1 | The number of threads of Netty TCP server to accept incoming connections and handle the traffic of accepted connections. | + + +### Example + +Before using the Netty source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "type": "tcp", + "host": "127.0.0.1", + "port": "10911", + "numberOfThreads": "1" + } + + ``` + +* YAML + + ```yaml + + configs: + type: "tcp" + host: "127.0.0.1" + port: 10999 + numberOfThreads: 1 + + ``` + +## Usage + +The following examples show how to use the Netty source connector with TCP and HTTP. + +### TCP + +1. Start Pulsar standalone. + + ```bash + + $ docker pull apachepulsar/pulsar:{version} + + $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standalone + + ``` + +2. Create a configuration file _netty-source-config.yaml_. + + ```yaml + + configs: + type: "tcp" + host: "127.0.0.1" + port: 10999 + numberOfThreads: 1 + + ``` + +3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server. + + ```bash + + $ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/ + + ``` + +4. Download the Netty source connector. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar + + ``` + +5. Start the Netty source connector. + + ```bash + + $ ./bin/pulsar-admin sources localrun \ + --archive pulsar-io-@pulsar:version@.nar \ + --tenant public \ + --namespace default \ + --name netty \ + --destination-topic-name netty-topic \ + --source-config-file netty-source-config.yaml \ + --parallelism 1 + + ``` + +6. Consume data. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + + $ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0 + + ``` + +7. Open another terminal window to send data to the Netty source. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + + $ apt-get update + + $ apt-get -y install telnet + + $ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999 + Trying 127.0.0.1... + Connected to 127.0.0.1. + Escape character is '^]'. + hello + world + + ``` + +8. The following information appears on the consumer terminal window. + + ```bash + + ----- got message ----- + hello + + ----- got message ----- + world + + ``` + +### HTTP + +1. Start Pulsar standalone. + + ```bash + + $ docker pull apachepulsar/pulsar:{version} + + $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standalone + + ``` + +2. Create a configuration file _netty-source-config.yaml_. + + ```yaml + + configs: + type: "http" + host: "127.0.0.1" + port: 10999 + numberOfThreads: 1 + + ``` + +3. Copy the configuration file _netty-source-config.yaml_ to Pulsar server. + + ```bash + + $ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/ + + ``` + +4. Download the Netty source connector. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar + + ``` + +5. Start the Netty source connector. + + ```bash + + $ ./bin/pulsar-admin sources localrun \ + --archive pulsar-io-@pulsar:version@.nar \ + --tenant public \ + --namespace default \ + --name netty \ + --destination-topic-name netty-topic \ + --source-config-file netty-source-config.yaml \ + --parallelism 1 + + ``` + +6. Consume data. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + + $ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0 + + ``` + +7. Open another terminal window to send data to the Netty source. + + ```bash + + $ docker exec -it pulsar-netty-standalone /bin/bash + + $ curl -X POST --data 'hello, world!' http://127.0.0.1:10999/ + + ``` + +8. The following information appears on the consumer terminal window. + + ```bash + + ----- got message ----- + hello, world! + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-nsq-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-nsq-source.md new file mode 100644 index 0000000000000..0bf16463b160b --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-nsq-source.md @@ -0,0 +1,25 @@ +--- +id: io-nsq-source +title: NSQ source connector +sidebar_label: "NSQ source connector" +original_id: io-nsq-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The NSQ source connector receives messages from NSQ topics +and writes messages to Pulsar topics. + +## Configuration + +The configuration of the NSQ source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `lookupds` |String| true | " " (empty string) | A comma-separated list of nsqlookupds to connect to. | +| `topic` | String|true | " " (empty string) | The NSQ topic to transport. | +| `channel` | String |false | pulsar-transport-{$topic} | The channel to consume from on the provided NSQ topic. | diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-overview.md b/site2/website-next/versioned_docs/version-2.7.2/io-overview.md index adb499818b0c9..3a55ff2aef2cc 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-overview.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-overview.md @@ -1,7 +1,7 @@ --- id: io-overview title: Pulsar connector overview -sidebar_label: Overview +sidebar_label: "Overview" original_id: io-overview --- diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md index 6df4de8a66903..67ca9710a10a8 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md @@ -1,7 +1,7 @@ --- id: io-quickstart title: How to connect Pulsar to database -sidebar_label: Get started +sidebar_label: "Get started" original_id: io-quickstart --- @@ -21,10 +21,8 @@ At the end of this tutorial, you are able to: :::tip - * These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. - * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. ::: @@ -39,74 +37,74 @@ For more information about **how to install a standalone Pulsar and built-in con 1. Start Pulsar locally. - ```bash - - bin/pulsar standalone - - ``` + ```bash + + bin/pulsar standalone + + ``` - All the components of a Pulsar service are start in order. - - You can curl those pulsar service endpoints to make sure Pulsar service is up running correctly. + All the components of a Pulsar service are start in order. + + You can curl those pulsar service endpoints to make sure Pulsar service is up running correctly. 2. Check Pulsar binary protocol port. - ```bash - - telnet localhost 6650 - - ``` + ```bash + + telnet localhost 6650 + + ``` 3. Check Pulsar Function cluster. - ```bash - - curl -s http://localhost:8080/admin/v2/worker/cluster - - ``` - - **Example output** - - ```json + ```bash + + curl -s http://localhost:8080/admin/v2/worker/cluster + + ``` - [{"workerId":"c-standalone-fw-localhost-6750","workerHostname":"localhost","port":6750}] + **Example output** - ``` + ```json + + [{"workerId":"c-standalone-fw-localhost-6750","workerHostname":"localhost","port":6750}] + + ``` 4. Make sure a public tenant and a default namespace exist. - ```bash - - curl -s http://localhost:8080/admin/v2/namespaces/public - - ``` - - **Example output** - - ```json + ```bash + + curl -s http://localhost:8080/admin/v2/namespaces/public + + ``` - ["public/default","public/functions"] + **Example output** - ``` + ```json + + ["public/default","public/functions"] + + ``` 5. All built-in connectors should be listed as available. - ```bash - - curl -s http://localhost:8080/admin/v2/functions/connectors - - ``` - - **Example output** - - ```json + ```bash + + curl -s http://localhost:8080/admin/v2/functions/connectors + + ``` - [{"name":"aerospike","description":"Aerospike database sink","sinkClass":"org.apache.pulsar.io.aerospike.AerospikeStringSink"},{"name":"cassandra","description":"Writes data into Cassandra","sinkClass":"org.apache.pulsar.io.cassandra.CassandraStringSink"},{"name":"kafka","description":"Kafka source and sink connector","sourceClass":"org.apache.pulsar.io.kafka.KafkaStringSource","sinkClass":"org.apache.pulsar.io.kafka.KafkaBytesSink"},{"name":"kinesis","description":"Kinesis sink connector","sinkClass":"org.apache.pulsar.io.kinesis.KinesisSink"},{"name":"rabbitmq","description":"RabbitMQ source connector","sourceClass":"org.apache.pulsar.io.rabbitmq.RabbitMQSource"},{"name":"twitter","description":"Ingest data from Twitter firehose","sourceClass":"org.apache.pulsar.io.twitter.TwitterFireHose"}] + **Example output** - ``` + ```json + + [{"name":"aerospike","description":"Aerospike database sink","sinkClass":"org.apache.pulsar.io.aerospike.AerospikeStringSink"},{"name":"cassandra","description":"Writes data into Cassandra","sinkClass":"org.apache.pulsar.io.cassandra.CassandraStringSink"},{"name":"kafka","description":"Kafka source and sink connector","sourceClass":"org.apache.pulsar.io.kafka.KafkaStringSource","sinkClass":"org.apache.pulsar.io.kafka.KafkaBytesSink"},{"name":"kinesis","description":"Kinesis sink connector","sinkClass":"org.apache.pulsar.io.kinesis.KinesisSink"},{"name":"rabbitmq","description":"RabbitMQ source connector","sourceClass":"org.apache.pulsar.io.rabbitmq.RabbitMQSource"},{"name":"twitter","description":"Ingest data from Twitter firehose","sourceClass":"org.apache.pulsar.io.twitter.TwitterFireHose"}] + + ``` - If an error occurs when starting Pulsar service, you may see an exception at the terminal running `pulsar/standalone`, - or you can navigate to the `logs` directory under the Pulsar directory to view the logs. + If an error occurs when starting Pulsar service, you may see an exception at the terminal running `pulsar/standalone`, + or you can navigate to the `logs` directory under the Pulsar directory to view the logs. ## Connect Pulsar to Cassandra @@ -114,9 +112,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip - * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). - * The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). ::: @@ -127,13 +123,13 @@ This example uses `cassandra` Docker image to start a single-node Cassandra clus 1. Start a Cassandra cluster. -```bash - -docker run -d --rm --name=cassandra -p 9042:9042 cassandra - -``` + ```bash + + docker run -d --rm --name=cassandra -p 9042:9042 cassandra + + ``` -:::note + :::note Before moving to the next steps, make sure the Cassandra cluster is running. @@ -141,68 +137,69 @@ docker run -d --rm --name=cassandra -p 9042:9042 cassandra 2. Make sure the Docker process is running. - ```bash - - docker ps - - ``` + ```bash + + docker ps + + ``` 3. Check the Cassandra logs to make sure the Cassandra process is running as expected. - ```bash - - docker logs cassandra - - ``` + ```bash + + docker logs cassandra + + ``` 4. Check the status of the Cassandra cluster. - ```bash - - docker exec cassandra nodetool status - - ``` - - **Example output** + ```bash + + docker exec cassandra nodetool status + + ``` - ``` - Datacenter: datacenter1 - ======================= - Status=Up/Down - |/ State=Normal/Leaving/Joining/Moving - -- Address Load Tokens Owns (effective) Host ID Rack - UN 172.17.0.2 103.67 KiB 256 100.0% af0e4b2f-84e0-4f0b-bb14-bd5f9070ff26 rack1 + **Example output** - ``` + ``` + + Datacenter: datacenter1 + ======================= + Status=Up/Down + |/ State=Normal/Leaving/Joining/Moving + -- Address Load Tokens Owns (effective) Host ID Rack + UN 172.17.0.2 103.67 KiB 256 100.0% af0e4b2f-84e0-4f0b-bb14-bd5f9070ff26 rack1 + + ``` 5. Use `cqlsh` to connect to the Cassandra cluster. - ```bash - - $ docker exec -ti cassandra cqlsh localhost - Connected to Test Cluster at localhost:9042. - [cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4] - Use HELP for help. - cqlsh> - - ``` + ```bash + + $ docker exec -ti cassandra cqlsh localhost + Connected to Test Cluster at localhost:9042. + [cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4] + Use HELP for help. + cqlsh> + + ``` 6. Create a keyspace `pulsar_test_keyspace`. - ```bash - - cqlsh> CREATE KEYSPACE pulsar_test_keyspace WITH replication = {'class':'SimpleStrategy', 'replication_factor':1}; - - ``` + ```bash + + cqlsh> CREATE KEYSPACE pulsar_test_keyspace WITH replication = {'class':'SimpleStrategy', 'replication_factor':1}; + + ``` 7. Create a table `pulsar_test_table`. - ```bash - - cqlsh> USE pulsar_test_keyspace; - cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY KEY, col text); - - ``` + ```bash + + cqlsh> USE pulsar_test_keyspace; + cqlsh:pulsar_test_keyspace> CREATE TABLE pulsar_test_table (key text PRIMARY KEY, col text); + + ``` ### Configure a Cassandra sink @@ -218,32 +215,31 @@ You can create a configuration file through one of the following methods. * JSON - ```json - - { - "roots": "localhost:9042", - "keyspace": "pulsar_test_keyspace", - "columnFamily": "pulsar_test_table", - "keyname": "key", - "columnName": "col" - } - - ``` - -* YAML + ```json + { + "roots": "localhost:9042", + "keyspace": "pulsar_test_keyspace", + "columnFamily": "pulsar_test_table", + "keyname": "key", + "columnName": "col" + } + + ``` - ```yaml - - configs: - roots: "localhost:9042" - keyspace: "pulsar_test_keyspace" - columnFamily: "pulsar_test_table" - keyname: "key" - columnName: "col" +* YAML - ``` + ```yaml + + configs: + roots: "localhost:9042" + keyspace: "pulsar_test_keyspace" + columnFamily: "pulsar_test_table" + keyname: "key" + columnName: "col" + ``` + For more information, see [Cassandra sink connector](io-cassandra-sink). ### Create a Cassandra sink @@ -281,160 +277,160 @@ to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. ```bash - + bin/pulsar-admin sinks get \ - --tenant public \ - --namespace default \ - --name cassandra-test-sink - + --tenant public \ + --namespace default \ + --name cassandra-test-sink + ``` **Example output** ```json - + { - "tenant": "public", - "namespace": "default", - "name": "cassandra-test-sink", - "className": "org.apache.pulsar.io.cassandra.CassandraStringSink", - "inputSpecs": { - "test_cassandra": { - "isRegexPattern": false - } - }, - "configs": { - "roots": "localhost:9042", - "keyspace": "pulsar_test_keyspace", - "columnFamily": "pulsar_test_table", - "keyname": "key", - "columnName": "col" - }, - "parallelism": 1, - "processingGuarantees": "ATLEAST_ONCE", - "retainOrdering": false, - "autoAck": true, - "archive": "builtin://cassandra" + "tenant": "public", + "namespace": "default", + "name": "cassandra-test-sink", + "className": "org.apache.pulsar.io.cassandra.CassandraStringSink", + "inputSpecs": { + "test_cassandra": { + "isRegexPattern": false + } + }, + "configs": { + "roots": "localhost:9042", + "keyspace": "pulsar_test_keyspace", + "columnFamily": "pulsar_test_table", + "keyname": "key", + "columnName": "col" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true, + "archive": "builtin://cassandra" } - + ``` * Check the status of a Cassandra sink. ```bash - + bin/pulsar-admin sinks status \ - --tenant public \ - --namespace default \ - --name cassandra-test-sink - + --tenant public \ + --namespace default \ + --name cassandra-test-sink + ``` **Example output** ```json - + { - "numInstances" : 1, - "numRunning" : 1, - "instances" : [ { - "instanceId" : 0, - "status" : { - "running" : true, - "error" : "", - "numRestarts" : 0, - "numReadFromPulsar" : 0, - "numSystemExceptions" : 0, - "latestSystemExceptions" : [ ], - "numSinkExceptions" : 0, - "latestSinkExceptions" : [ ], - "numWrittenToSink" : 0, - "lastReceivedTime" : 0, - "workerId" : "c-standalone-fw-localhost-8080" - } - } ] + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 0, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 0, + "lastReceivedTime" : 0, + "workerId" : "c-standalone-fw-localhost-8080" + } + } ] } - + ``` ### Verify a Cassandra sink 1. Produce some messages to the input topic of the Cassandra sink _test_cassandra_. - ```bash + ```bash + + for i in {0..9}; do bin/pulsar-client produce -m "key-$i" -n 1 test_cassandra; done + + ``` - for i in {0..9}; do bin/pulsar-client produce -m "key-$i" -n 1 test_cassandra; done +2. Inspect the status of the Cassandra sink _test_cassandra_. - ``` + ```bash + + bin/pulsar-admin sinks status \ + --tenant public \ + --namespace default \ + --name cassandra-test-sink + + ``` -2. Inspect the status of the Cassandra sink _test_cassandra_. + You can see 10 messages are processed by the Cassandra sink _test_cassandra_. - ```bash - - bin/pulsar-admin sinks status \ - --tenant public \ - --namespace default \ - --name cassandra-test-sink - - ``` - - You can see 10 messages are processed by the Cassandra sink _test_cassandra_. - - **Example output** - - ```json - - { - "numInstances" : 1, - "numRunning" : 1, - "instances" : [ { - "instanceId" : 0, - "status" : { - "running" : true, - "error" : "", - "numRestarts" : 0, - "numReadFromPulsar" : 10, - "numSystemExceptions" : 0, - "latestSystemExceptions" : [ ], - "numSinkExceptions" : 0, - "latestSinkExceptions" : [ ], - "numWrittenToSink" : 10, - "lastReceivedTime" : 1551685489136, - "workerId" : "c-standalone-fw-localhost-8080" - } - } ] - } + **Example output** - ``` + ```json + + { + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 10, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 10, + "lastReceivedTime" : 1551685489136, + "workerId" : "c-standalone-fw-localhost-8080" + } + } ] + } + + ``` 3. Use `cqlsh` to connect to the Cassandra cluster. ```bash - + docker exec -ti cassandra cqlsh localhost - + ``` 4. Check the data of the Cassandra table _pulsar_test_table_. ```bash - + cqlsh> use pulsar_test_keyspace; cqlsh:pulsar_test_keyspace> select * from pulsar_test_table; key | col --------+-------- - key-5 | key-5 - key-0 | key-0 - key-9 | key-9 - key-2 | key-2 - key-1 | key-1 - key-3 | key-3 - key-6 | key-6 - key-7 | key-7 - key-4 | key-4 - key-8 | key-8 - + key-5 | key-5 + key-0 | key-0 + key-9 | key-9 + key-2 | key-2 + key-1 | key-1 + key-3 | key-3 + key-6 | key-6 + key-7 | key-7 + key-4 | key-4 + key-8 | key-8 + ``` ### Delete a Cassandra Sink @@ -457,12 +453,11 @@ This section demonstrates how to connect Pulsar to PostgreSQL. :::tip - * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). - * The JDBC sink connector pulls messages from Pulsar topics ::: + and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. >For more information, see [JDBC sink connector](io-jdbc-sink). @@ -473,35 +468,35 @@ This example uses the PostgreSQL 12 docker image to start a single-node PostgreS 1. Pull the PostgreSQL 12 image from Docker. - ```bash - - $ docker pull postgres:12 - - ``` + ```bash + + $ docker pull postgres:12 + + ``` 2. Start PostgreSQL. - ```bash - - $ docker run -d -it --rm \ - --name pulsar-postgres \ - -p 5432:5432 \ - -e POSTGRES_PASSWORD=password \ - -e POSTGRES_USER=postgres \ - postgres:12 - - ``` + ```bash + + $ docker run -d -it --rm \ + --name pulsar-postgres \ + -p 5432:5432 \ + -e POSTGRES_PASSWORD=password \ + -e POSTGRES_USER=postgres \ + postgres:12 + + ``` - #### Tip - - Flag | Description | This example - ---|---|---| - `-d` | To start a container in detached mode. | / - `-it` | Keep STDIN open even if not attached and allocate a terminal. | / - `--rm` | Remove the container automatically when it exits. | / - `-name` | Assign a name to the container. | This example specifies _pulsar-postgres_ for the container. - `-p` | Publish the port of the container to the host. | This example publishes the port _5432_ of the container to the host. - `-e` | Set environment variables. | This example sets the following variables:
    - The password for the user is _password_.
    - The name for the user is _postgres_. + #### Tip + + Flag | Description | This example + ---|---|---| + `-d` | To start a container in detached mode. | / + `-it` | Keep STDIN open even if not attached and allocate a terminal. | / + `--rm` | Remove the container automatically when it exits. | / + `-name` | Assign a name to the container. | This example specifies _pulsar-postgres_ for the container. + `-p` | Publish the port of the container to the host. | This example publishes the port _5432_ of the container to the host. + `-e` | Set environment variables. | This example sets the following variables:
    - The password for the user is _password_.
    - The name for the user is _postgres_. :::tip @@ -511,46 +506,46 @@ This example uses the PostgreSQL 12 docker image to start a single-node PostgreS 3. Check if PostgreSQL has been started successfully. - ```bash - - $ docker logs -f pulsar-postgres - - ``` - - PostgreSQL has been started successfully if the following message appears. - - ```text + ```bash + + $ docker logs -f pulsar-postgres + + ``` - 2020-05-11 20:09:24.492 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit - 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 - 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv6 address "::", port 5432 - 2020-05-11 20:09:24.499 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" - 2020-05-11 20:09:24.523 UTC [55] LOG: database system was shut down at 2020-05-11 20:09:24 UTC - 2020-05-11 20:09:24.533 UTC [1] LOG: database system is ready to accept connections + PostgreSQL has been started successfully if the following message appears. - ``` + ```text + + 2020-05-11 20:09:24.492 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit + 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 + 2020-05-11 20:09:24.492 UTC [1] LOG: listening on IPv6 address "::", port 5432 + 2020-05-11 20:09:24.499 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" + 2020-05-11 20:09:24.523 UTC [55] LOG: database system was shut down at 2020-05-11 20:09:24 UTC + 2020-05-11 20:09:24.533 UTC [1] LOG: database system is ready to accept connections + + ``` 4. Access to PostgreSQL. - ```bash - - $ docker exec -it pulsar-postgres /bin/bash - - ``` + ```bash + + $ docker exec -it pulsar-postgres /bin/bash + + ``` 5. Create a PostgreSQL table _pulsar_postgres_jdbc_sink_. - ```bash - - $ psql -U postgres postgres - - postgres=# create table if not exists pulsar_postgres_jdbc_sink - ( - id serial PRIMARY KEY, - name VARCHAR(255) NOT NULL - ); - - ``` + ```bash + + $ psql -U postgres postgres + + postgres=# create table if not exists pulsar_postgres_jdbc_sink + ( + id serial PRIMARY KEY, + name VARCHAR(255) NOT NULL + ); + + ``` ### Configure a JDBC sink @@ -560,35 +555,35 @@ In this section, you need to configure a JDBC sink connector. 1. Add a configuration file. - To run a JDBC sink connector, you need to prepare a YAML configuration file including the information that Pulsar connector runtime needs to know. - - For example, how Pulsar connector can find the PostgreSQL cluster, what is the JDBC URL and the table that Pulsar connector uses for writing messages to. - - Create a _pulsar-postgres-jdbc-sink.yaml_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. - - ```yaml + To run a JDBC sink connector, you need to prepare a YAML configuration file including the information that Pulsar connector runtime needs to know. + + For example, how Pulsar connector can find the PostgreSQL cluster, what is the JDBC URL and the table that Pulsar connector uses for writing messages to. - configs: - userName: "postgres" - password: "password" - jdbcUrl: "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink" - tableName: "pulsar_postgres_jdbc_sink" + Create a _pulsar-postgres-jdbc-sink.yaml_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. - ``` + ```yaml + + configs: + userName: "postgres" + password: "password" + jdbcUrl: "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink" + tableName: "pulsar_postgres_jdbc_sink" + + ``` 2. Create a schema. - Create a _avro-schema_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. - - ```json + Create a _avro-schema_ file, copy the following contents to this file, and place the file in the `pulsar/connectors` folder. - { - "type": "AVRO", - "schema": "{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}", - "properties": {} - } - - ``` + ```json + + { + "type": "AVRO", + "schema": "{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}", + "properties": {} + } + + ``` :::tip @@ -598,29 +593,29 @@ In this section, you need to configure a JDBC sink connector. 3. Upload a schema to a topic. - This example uploads the _avro-schema_ schema to the _pulsar-postgres-jdbc-sink-topic_ topic. - - ```bash - - $ bin/pulsar-admin schemas upload pulsar-postgres-jdbc-sink-topic -f ./connectors/avro-schema + This example uploads the _avro-schema_ schema to the _pulsar-postgres-jdbc-sink-topic_ topic. - ``` + ```bash + + $ bin/pulsar-admin schemas upload pulsar-postgres-jdbc-sink-topic -f ./connectors/avro-schema + + ``` 4. Check if the schema has been uploaded successfully. - ```bash - - $ bin/pulsar-admin schemas get pulsar-postgres-jdbc-sink-topic - - ``` - - The schema has been uploaded successfully if the following message appears. - - ```json + ```bash + + $ bin/pulsar-admin schemas get pulsar-postgres-jdbc-sink-topic + + ``` - {"name":"pulsar-postgres-jdbc-sink-topic","schema":"{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}","type":"AVRO","properties":{}} + The schema has been uploaded successfully if the following message appears. - ``` + ```json + + {"name":"pulsar-postgres-jdbc-sink-topic","schema":"{\"type\":\"record\",\"name\":\"Test\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"int\"]},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}","type":"AVRO","properties":{}} + + ``` ### Create a JDBC sink @@ -656,7 +651,6 @@ This sink connector runs as a Pulsar Function and writes the messages produced i :::tip - For more information about `pulsar-admin sinks create options`, see [here](io-cli.md#sinks). ::: @@ -677,119 +671,116 @@ to monitor a connector and perform other operations on it. * List all running JDBC sink(s). ```bash - + $ bin/pulsar-admin sinks list \ --tenant public \ --namespace default - + ``` -:::tip - + :::tip -For more information about `pulsar-admin sinks list options`, see [here](io-cli.md/#list-1). + For more information about `pulsar-admin sinks list options`, see [here](io-cli.md/#list-1). -::: + ::: The result shows that only the _postgres-jdbc-sink_ sink is running. ```json - + [ "pulsar-postgres-jdbc-sink" ] - + ``` * Get the information of a JDBC sink. ```bash - + $ bin/pulsar-admin sinks get \ --tenant public \ --namespace default \ --name pulsar-postgres-jdbc-sink - + ``` -:::tip + :::tip + For more information about `pulsar-admin sinks get options`, see [here](io-cli.md/#get-1). -For more information about `pulsar-admin sinks get options`, see [here](io-cli.md/#get-1). - -::: + ::: The result shows the information of the sink connector, including tenant, namespace, topic and so on. ```json - + { - "tenant": "public", - "namespace": "default", - "name": "pulsar-postgres-jdbc-sink", - "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink", - "inputSpecs": { - "pulsar-postgres-jdbc-sink-topic": { - "isRegexPattern": false - } - }, - "configs": { - "password": "password", - "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", - "userName": "postgres", - "tableName": "pulsar_postgres_jdbc_sink" - }, - "parallelism": 1, - "processingGuarantees": "ATLEAST_ONCE", - "retainOrdering": false, - "autoAck": true + "tenant": "public", + "namespace": "default", + "name": "pulsar-postgres-jdbc-sink", + "className": "org.apache.pulsar.io.jdbc.PostgresJdbcAutoSchemaSink", + "inputSpecs": { + "pulsar-postgres-jdbc-sink-topic": { + "isRegexPattern": false + } + }, + "configs": { + "password": "password", + "jdbcUrl": "jdbc:postgresql://localhost:5432/pulsar_postgres_jdbc_sink", + "userName": "postgres", + "tableName": "pulsar_postgres_jdbc_sink" + }, + "parallelism": 1, + "processingGuarantees": "ATLEAST_ONCE", + "retainOrdering": false, + "autoAck": true } - + ``` * Get the status of a JDBC sink ```bash - + $ bin/pulsar-admin sinks status \ --tenant public \ --namespace default \ --name pulsar-postgres-jdbc-sink - + ``` -:::tip + :::tip + For more information about `pulsar-admin sinks status options`, see [here](io-cli.md/#status-1). -For more information about `pulsar-admin sinks status options`, see [here](io-cli.md/#status-1). - -::: + ::: The result shows the current status of sink connector, including the number of instance, running status, worker ID and so on. ```json - + { - "numInstances" : 1, - "numRunning" : 1, - "instances" : [ { - "instanceId" : 0, - "status" : { - "running" : true, - "error" : "", - "numRestarts" : 0, - "numReadFromPulsar" : 0, - "numSystemExceptions" : 0, - "latestSystemExceptions" : [ ], - "numSinkExceptions" : 0, - "latestSinkExceptions" : [ ], - "numWrittenToSink" : 0, - "lastReceivedTime" : 0, - "workerId" : "c-standalone-fw-192.168.2.52-8080" - } - } ] + "numInstances" : 1, + "numRunning" : 1, + "instances" : [ { + "instanceId" : 0, + "status" : { + "running" : true, + "error" : "", + "numRestarts" : 0, + "numReadFromPulsar" : 0, + "numSystemExceptions" : 0, + "latestSystemExceptions" : [ ], + "numSinkExceptions" : 0, + "latestSinkExceptions" : [ ], + "numWrittenToSink" : 0, + "lastReceivedTime" : 0, + "workerId" : "c-standalone-fw-192.168.2.52-8080" + } + } ] } - + ``` ### Stop a JDBC sink @@ -808,7 +799,6 @@ $ bin/pulsar-admin sinks stop \ :::tip - For more information about `pulsar-admin sinks stop options`, see [here](io-cli.md/#stop-1). ::: @@ -831,13 +821,12 @@ to restart a connector and perform other operations on it. $ bin/pulsar-admin sinks restart \ --tenant public \ --namespace default \ ---name pulsar-postgres-jdbc-sink +--name pulsar-postgres-jdbc-sink ``` :::tip - For more information about `pulsar-admin sinks restart options`, see [here](io-cli.md/#restart-1). ::: @@ -852,11 +841,8 @@ The sink instance has been started successfully if the following message disappe :::tip - * Optionally, you can run a standalone sink connector using `pulsar-admin sinks localrun options`. - Note that `pulsar-admin sinks localrun options` **runs a sink connector locally**, while `pulsar-admin sinks start options` **starts a sink connector in a cluster**. - * For more information about `pulsar-admin sinks localrun options`, see [here](io-cli.md#localrun-1). ::: @@ -878,7 +864,6 @@ $ bin/pulsar-admin sinks update \ :::tip - For more information about `pulsar-admin sinks update options`, see [here](io-cli.md/#update-1). ::: @@ -948,7 +933,6 @@ $ bin/pulsar-admin sinks delete \ :::tip - For more information about `pulsar-admin sinks delete options`, see [here](io-cli.md/#delete-1). ::: @@ -981,3 +965,4 @@ HTTP 404 Not Found Reason: Sink pulsar-postgres-jdbc-sink doesn't exist ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-sink.md new file mode 100644 index 0000000000000..48f1c75678559 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-sink.md @@ -0,0 +1,89 @@ +--- +id: io-rabbitmq-sink +title: RabbitMQ sink connector +sidebar_label: "RabbitMQ sink connector" +original_id: io-rabbitmq-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The RabbitMQ sink connector pulls messages from Pulsar topics +and persist the messages to RabbitMQ queues. + + +## Configuration + +The configuration of the RabbitMQ sink connector has the following properties. + + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `connectionName` |String| true | " " (empty string) | The connection name. | +| `host` | String| true | " " (empty string) | The RabbitMQ host. | +| `port` | int |true | 5672 | The RabbitMQ port. | +| `virtualHost` |String|true | / | The virtual host used to connect to RabbitMQ. | +| `username` | String|false | guest | The username used to authenticate to RabbitMQ. | +| `password` | String|false | guest | The password used to authenticate to RabbitMQ. | +| `queueName` | String|true | " " (empty string) | The RabbitMQ queue name that messages should be read from or written to. | +| `requestedChannelMax` | int|false | 0 | The initially requested maximum channel number.

    0 means unlimited. | +| `requestedFrameMax` | int|false |0 | The initially requested maximum frame size in octets.

    0 means unlimited. | +| `connectionTimeout` | int|false | 60000 | The timeout of TCP connection establishment in milliseconds.

    0 means infinite. | +| `handshakeTimeout` | int|false | 10000 | The timeout of AMQP0-9-1 protocol handshake in milliseconds. | +| `requestedHeartbeat` | int|false | 60 | The exchange to publish messages. | +| `exchangeName` | String|true | " " (empty string) | The maximum number of messages that the server delivers.

    0 means unlimited. | +| `prefetchGlobal` |String|true | " " (empty string) |The routing key used to publish messages. | + + +### Example + +Before using the RabbitMQ sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "host": "localhost", + "port": "5672", + "virtualHost": "/", + "username": "guest", + "password": "guest", + "queueName": "test-queue", + "connectionName": "test-connection", + "requestedChannelMax": "0", + "requestedFrameMax": "0", + "connectionTimeout": "60000", + "handshakeTimeout": "10000", + "requestedHeartbeat": "60", + "exchangeName": "test-exchange", + "routingKey": "test-key" + } + + ``` + +* YAML + + ```yaml + + configs: + host: "localhost" + port: 5672 + virtualHost: "/", + username: "guest" + password: "guest" + queueName: "test-queue" + connectionName: "test-connection" + requestedChannelMax: 0 + requestedFrameMax: 0 + connectionTimeout: 60000 + handshakeTimeout: 10000 + requestedHeartbeat: 60 + exchangeName: "test-exchange" + routingKey: "test-key" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-source.md new file mode 100644 index 0000000000000..0a10cfaab1b1a --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-rabbitmq-source.md @@ -0,0 +1,89 @@ +--- +id: io-rabbitmq-source +title: RabbitMQ source connector +sidebar_label: "RabbitMQ source connector" +original_id: io-rabbitmq-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The RabbitMQ source connector receives messages from RabbitMQ clusters +and writes messages to Pulsar topics. + +## Configuration + +The configuration of the RabbitMQ source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `connectionName` |String| true | " " (empty string) | The connection name. | +| `host` | String| true | " " (empty string) | The RabbitMQ host. | +| `port` | int |true | 5672 | The RabbitMQ port. | +| `virtualHost` |String|true | / | The virtual host used to connect to RabbitMQ. | +| `username` | String|false | guest | The username used to authenticate to RabbitMQ. | +| `password` | String|false | guest | The password used to authenticate to RabbitMQ. | +| `queueName` | String|true | " " (empty string) | The RabbitMQ queue name that messages should be read from or written to. | +| `requestedChannelMax` | int|false | 0 | The initially requested maximum channel number.

    0 means unlimited. | +| `requestedFrameMax` | int|false |0 | The initially requested maximum frame size in octets.

    0 means unlimited. | +| `connectionTimeout` | int|false | 60000 | The timeout of TCP connection establishment in milliseconds.

    0 means infinite. | +| `handshakeTimeout` | int|false | 10000 | The timeout of AMQP0-9-1 protocol handshake in milliseconds. | +| `requestedHeartbeat` | int|false | 60 | The requested heartbeat timeout in seconds. | +| `prefetchCount` | int|false | 0 | The maximum number of messages that the server delivers.

    0 means unlimited. | +| `prefetchGlobal` | boolean|false | false |Whether the setting should be applied to the entire channel rather than each consumer. | +| `passive` | boolean|false | false | Whether the rabbitmq consumer should create its own queue or bind to an existing one. | + +### Example + +Before using the RabbitMQ source connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "host": "localhost", + "port": "5672", + "virtualHost": "/", + "username": "guest", + "password": "guest", + "queueName": "test-queue", + "connectionName": "test-connection", + "requestedChannelMax": "0", + "requestedFrameMax": "0", + "connectionTimeout": "60000", + "handshakeTimeout": "10000", + "requestedHeartbeat": "60", + "prefetchCount": "0", + "prefetchGlobal": "false", + "passive": "false" + } + + ``` + +* YAML + + ```yaml + + configs: + host: "localhost" + port: 5672 + virtualHost: "/" + username: "guest" + password: "guest" + queueName: "test-queue" + connectionName: "test-connection" + requestedChannelMax: 0 + requestedFrameMax: 0 + connectionTimeout: 60000 + handshakeTimeout: 10000 + requestedHeartbeat: 60 + prefetchCount: 0 + prefetchGlobal: "false" + passive: "false" + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-redis-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-redis-sink.md new file mode 100644 index 0000000000000..49eb119802736 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-redis-sink.md @@ -0,0 +1,78 @@ +--- +id: io-redis-sink +title: Redis sink connector +sidebar_label: "Redis sink connector" +original_id: io-redis-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Redis sink connector pulls messages from Pulsar topics +and persists the messages to a Redis database. + + + +## Configuration + +The configuration of the Redis sink connector has the following properties. + + + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `redisHosts` |String|true|" " (empty string) | A comma-separated list of Redis hosts to connect to. | +| `redisPassword` |String|false|" " (empty string) | The password used to connect to Redis. | +| `redisDatabase` | int|true|0 | The Redis database to connect to. | +| `clientMode` |String| false|Standalone | The client mode when interacting with Redis cluster.

    Below are the available options:
  • Standalone
  • Cluster
    • | +| `autoReconnect` | boolean|false|true | Whether the Redis client automatically reconnect or not. | +| `requestQueue` | int|false|2147483647 | The maximum number of queued requests to Redis. | +| `tcpNoDelay` |boolean| false| false | Whether to enable TCP with no delay or not. | +| `keepAlive` | boolean|false | false |Whether to enable a keepalive to Redis or not. | +| `connectTimeout` |long| false|10000 | The time to wait before timing out when connecting in milliseconds. | +| `operationTimeout` | long|false|10000 | The time before an operation is marked as timed out in milliseconds . | +| `batchTimeMs` | int|false|1000 | The Redis operation time in milliseconds. | +| `batchSize` | int|false|200 | The batch size of writing to Redis database. | + + +### Example + +Before using the Redis sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "redisHosts": "localhost:6379", + "redisPassword": "fake@123", + "redisDatabase": "1", + "clientMode": "Standalone", + "operationTimeout": "2000", + "batchSize": "100", + "batchTimeMs": "1000", + "connectTimeout": "3000" + } + + ``` + +* YAML + + ```yaml + + { + redisHosts: "localhost:6379" + redisPassword: "fake@123" + redisDatabase: 1 + clientMode: "Standalone" + operationTimeout: 2000 + batchSize: 100 + batchTimeMs: 1000 + connectTimeout: 3000 + } + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-solr-sink.md b/site2/website-next/versioned_docs/version-2.7.2/io-solr-sink.md new file mode 100644 index 0000000000000..d7b31ad97cde8 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-solr-sink.md @@ -0,0 +1,69 @@ +--- +id: io-solr-sink +title: Solr sink connector +sidebar_label: "Solr sink connector" +original_id: io-solr-sink +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Solr sink connector pulls messages from Pulsar topics +and persists the messages to Solr collections. + + + +## Configuration + +The configuration of the Solr sink connector has the following properties. + + + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `solrUrl` | String|true|" " (empty string) |
  • Comma-separated zookeeper hosts with chroot used in the SolrCloud mode.
    **Example**
    `localhost:2181,localhost:2182/chroot`

  • URL to connect to Solr used in standalone mode.
    **Example**
    `localhost:8983/solr`
    • | +| `solrMode` | String|true|SolrCloud| The client mode when interacting with the Solr cluster.

    Below are the available options:
  • Standalone
  • SolrCloud
    • | +| `solrCollection` |String|true| " " (empty string) | Solr collection name to which records need to be written. | +| `solrCommitWithinMs` |int| false|10 | The time within million seconds for Solr updating commits.| +| `username` |String|false| " " (empty string) | The username for basic authentication.

    **Note: `usename` is case-sensitive.** | +| `password` | String|false| " " (empty string) | The password for basic authentication.

    **Note: `password` is case-sensitive.** | + + + +### Example + +Before using the Solr sink connector, you need to create a configuration file through one of the following methods. + +* JSON + + ```json + + { + "solrUrl": "localhost:2181,localhost:2182/chroot", + "solrMode": "SolrCloud", + "solrCollection": "techproducts", + "solrCommitWithinMs": 100, + "username": "fakeuser", + "password": "fake@123" + } + + ``` + +* YAML + + ```yaml + + { + solrUrl: "localhost:2181,localhost:2182/chroot" + solrMode: "SolrCloud" + solrCollection: "techproducts" + solrCommitWithinMs: 100 + username: "fakeuser" + password: "fake@123" + } + + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-twitter-source.md b/site2/website-next/versioned_docs/version-2.7.2/io-twitter-source.md new file mode 100644 index 0000000000000..101602e246fb8 --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-twitter-source.md @@ -0,0 +1,32 @@ +--- +id: io-twitter-source +title: Twitter Firehose source connector +sidebar_label: "Twitter Firehose source connector" +original_id: io-twitter-source +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + +The Twitter Firehose source connector receives tweets from Twitter Firehose and +writes the tweets to Pulsar topics. + +## Configuration + +The configuration of the Twitter Firehose source connector has the following properties. + +### Property + +| Name | Type|Required | Default | Description +|------|----------|----------|---------|-------------| +| `consumerKey` | String|true | " " (empty string) | The twitter OAuth consumer key.

    For more information, see [Access tokens](https://developer.twitter.com/en/docs/basics/authentication/guides/access-tokens). | +| `consumerSecret` | String |true | " " (empty string) | The twitter OAuth consumer secret. | +| `token` | String|true | " " (empty string) | The twitter OAuth token. | +| `tokenSecret` | String|true | " " (empty string) | The twitter OAuth secret. | +| `guestimateTweetTime`|Boolean|false|false|Most firehose events have null createdAt time.

    If `guestimateTweetTime` set to true, the connector estimates the createdTime of each firehose event to be current time. +| `clientName` | String |false | openconnector-twitter-source| The twitter firehose client name. | +| `clientHosts` |String| false | Constants.STREAM_HOST | The twitter firehose hosts to which client connects. | +| `clientBufferSize` | int|false | 50000 | The buffer size for buffering tweets fetched from twitter firehose. | + +> For more information about OAuth credentials, see [Twitter developers portal](https://developer.twitter.com/en.html). diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-twitter.md b/site2/website-next/versioned_docs/version-2.7.2/io-twitter.md new file mode 100644 index 0000000000000..53f949863702d --- /dev/null +++ b/site2/website-next/versioned_docs/version-2.7.2/io-twitter.md @@ -0,0 +1,11 @@ +--- +id: io-twitter +title: Twitter Firehose Connector +sidebar_label: "Twitter Firehose Connector" +original_id: io-twitter +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-use.md b/site2/website-next/versioned_docs/version-2.7.2/io-use.md index 1c98a98227d7a..43b7a30cadc33 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-use.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-use.md @@ -1,7 +1,7 @@ --- id: io-use title: How to use Pulsar connectors -sidebar_label: Use +sidebar_label: "Use" original_id: io-use --- @@ -17,7 +17,6 @@ Pulsar bundles several [builtin connectors](io-connectors) used to move data in :::note - When using a non-builtin connector, you need to specify the path of a archive file for the connector. ::: @@ -44,6 +43,7 @@ To configure a default folder for builtin connectors, set the `connectorsDirecto Set the `./connectors` folder as the default storage location for builtin connectors. ``` + ######################## # Connectors ######################## @@ -213,6 +213,7 @@ Create a source connector. Use the `create` subcommand. ``` + $ pulsar-admin sources create options ``` @@ -229,60 +230,60 @@ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/source * Create a source connector with a **local file**. - ```java - - void createSource(SourceConfig sourceConfig, - String fileName) - throws PulsarAdminException - - ``` + ```java + + void createSource(SourceConfig sourceConfig, + String fileName) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - |Name|Description - |---|--- - `sourceConfig` | The source configuration object + |Name|Description + |---|--- + `sourceConfig` | The source configuration object **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`createSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSource-SourceConfig-java.lang.String-). + For more information, see [`createSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSource-SourceConfig-java.lang.String-). * Create a source connector using a **remote file** with a URL from which fun-pkg can be downloaded. - ```java - - void createSourceWithUrl(SourceConfig sourceConfig, - String pkgUrl) - throws PulsarAdminException - - ``` + ```java + + void createSourceWithUrl(SourceConfig sourceConfig, + String pkgUrl) + throws PulsarAdminException + + ``` - Supported URLs are `http` and `file`. + Supported URLs are `http` and `file`. - **Example** + **Example** - * HTTP: http://www.repo.com/fileName.jar + * HTTP: http://www.repo.com/fileName.jar - * File: file:///dir/fileName.jar + * File: file:///dir/fileName.jar - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `sourceConfig` | The source configuration object - `pkgUrl` | URL from which pkg can be downloaded + Parameter| Description + |---|--- + `sourceConfig` | The source configuration object + `pkgUrl` | URL from which pkg can be downloaded - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSourceWithUrl-SourceConfig-java.lang.String-). + For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#createSourceWithUrl-SourceConfig-java.lang.String-). @@ -314,6 +315,7 @@ Create a sink connector. Use the `create` subcommand. ``` + $ pulsar-admin sinks create options ``` @@ -329,62 +331,61 @@ Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/ * Create a sink connector with a **local file**. - - - ```java - - void createSink(SinkConfig sinkConfig, - String fileName) - throws PulsarAdminException - ``` + ```java + + void createSink(SinkConfig sinkConfig, + String fileName) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - |Name|Description - |---|--- - `sinkConfig` | The sink configuration object + |Name|Description + |---|--- + `sinkConfig` | The sink configuration object - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`createSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSink-SinkConfig-java.lang.String-). + For more information, see [`createSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSink-SinkConfig-java.lang.String-). * Create a sink connector using a **remote file** with a URL from which fun-pkg can be downloaded. - ```java - - void createSinkWithUrl(SinkConfig sinkConfig, - String pkgUrl) - throws PulsarAdminException - - ``` + ```java + + void createSinkWithUrl(SinkConfig sinkConfig, + String pkgUrl) + throws PulsarAdminException + + ``` - Supported URLs are `http` and `file`. + Supported URLs are `http` and `file`. - **Example** + **Example** - * HTTP: http://www.repo.com/fileName.jar + * HTTP: http://www.repo.com/fileName.jar - * File: file:///dir/fileName.jar + * File: file:///dir/fileName.jar - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `sinkConfig` | The sink configuration object - `pkgUrl` | URL from which pkg can be downloaded + Parameter| Description + |---|--- + `sinkConfig` | The sink configuration object + `pkgUrl` | URL from which pkg can be downloaded - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`createSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSinkWithUrl-SinkConfig-java.lang.String-). + For more information, see [`createSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#createSinkWithUrl-SinkConfig-java.lang.String-). @@ -416,6 +417,7 @@ Start a source connector. Use the `start` subcommand. ``` + $ pulsar-admin sources start options ``` @@ -427,11 +429,11 @@ For more information, see [here](io-cli.md#start). * Start **all** source connectors. - Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/start|operation/startSource?version=@pulsar:version_number@} + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/start|operation/startSource?version=@pulsar:version_number@} * Start a **specified** source connector. - Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSource?version=@pulsar:version_number@} + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSource?version=@pulsar:version_number@} @@ -459,6 +461,7 @@ Start a sink connector. Use the `start` subcommand. ``` + $ pulsar-admin sinks start options ``` @@ -470,11 +473,11 @@ For more information, see [here](io-cli.md#start-1). * Start **all** sink connectors. - Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/start|operation/startSink?version=@pulsar:version_number@} + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sources/:tenant/:namespace/:sinkName/start|operation/startSink?version=@pulsar:version_number@} * Start a **specified** sink connector. - Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSink?version=@pulsar:version_number@} + Send a `POST` request to this endpoint: {@inject: endpoint|POST|/admin/v3/sinks/:tenant/:namespace/:sourceName/:instanceId/start|operation/startSink?version=@pulsar:version_number@} @@ -502,6 +505,7 @@ Run a source connector locally. Use the `localrun` subcommand. ``` + $ pulsar-admin sources localrun options ``` @@ -530,6 +534,7 @@ Run a sink connector locally. Use the `localrun` subcommand. ``` + $ pulsar-admin sinks localrun options ``` @@ -580,6 +585,7 @@ Get the information of a source connector. Use the `get` subcommand. ``` + $ pulsar-admin sources get options ``` @@ -630,6 +636,7 @@ This is a sourceConfig. This is a sourceConfig example. ``` + { "tenant": "public", "namespace": "default", @@ -702,6 +709,7 @@ Get the information of a sink connector. Use the `get` subcommand. ``` + $ pulsar-admin sinks get options ``` @@ -822,6 +830,7 @@ Get the list of all running source connectors. Use the `list` subcommand. ``` + $ pulsar-admin sources list options ``` @@ -846,7 +855,9 @@ List listSources(String tenant, **Response example** -```java ["f1", "f2", "f3"] +```java + +["f1", "f2", "f3"] ``` @@ -889,6 +900,7 @@ Get the list of all running sink connectors. Use the `list` subcommand. ``` + $ pulsar-admin sinks list options ``` @@ -913,7 +925,9 @@ List listSinks(String tenant, **Response example** -```java ["f1", "f2", "f3"] +```java + +["f1", "f2", "f3"] ``` @@ -960,6 +974,7 @@ Get the current status of a source connector. Use the `status` subcommand. ``` + $ pulsar-admin sources status options ``` @@ -982,59 +997,59 @@ For more information, see [here](io-cli.md#status). * Get the current status of **all** source connectors. - ```java - - SourceStatus getSourceStatus(String tenant, - String namespace, - String source) - throws PulsarAdminException - - ``` + ```java + + SourceStatus getSourceStatus(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `sink` | Source name + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name - **Exception** + **Exception** - Name | Description - |---|--- - `PulsarAdminException` | Unexpected error + Name | Description + |---|--- + `PulsarAdminException` | Unexpected error - For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSource-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSource-java.lang.String-java.lang.String-java.lang.String-). * Gets the current status of a **specified** source connector. - ```java - - SourceStatus.SourceInstanceStatus.SourceInstanceStatusData getSourceStatus(String tenant, - String namespace, - String source, - int id) - throws PulsarAdminException - - ``` + ```java + + SourceStatus.SourceInstanceStatus.SourceInstanceStatusData getSourceStatus(String tenant, + String namespace, + String source, + int id) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `sink` | Source name - `id` | Source instanceID + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + `id` | Source instanceID - **Exception** + **Exception** - Exception name | Description - |---|--- - `PulsarAdminException` | Unexpected error + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error - For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSourceStatus-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`getSourceStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#getSourceStatus-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1066,6 +1081,7 @@ Get the current status of a Pulsar sink connector. Use the `status` subcommand. ``` + $ pulsar-admin sinks status options ``` @@ -1088,59 +1104,59 @@ For more information, see [here](io-cli.md#status-1). * Get the current status of **all** sink connectors. - ```java - - SinkStatus getSinkStatus(String tenant, - String namespace, - String sink) - throws PulsarAdminException - - ``` + ```java + + SinkStatus getSinkStatus(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `sink` | Source name + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name - **Exception** + **Exception** - Exception name | Description - |---|--- - `PulsarAdminException` | Unexpected error + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error - For more information, see [`getSinkStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`getSinkStatus`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-). * Gets the current status of a **specified** source connector. - ```java - - SinkStatus.SinkInstanceStatus.SinkInstanceStatusData getSinkStatus(String tenant, - String namespace, - String sink, - int id) - throws PulsarAdminException - - ``` + ```java + + SinkStatus.SinkInstanceStatus.SinkInstanceStatusData getSinkStatus(String tenant, + String namespace, + String sink, + int id) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - Parameter| Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `sink` | Source name - `id` | Sink instanceID + Parameter| Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Source name + `id` | Sink instanceID - **Exception** + **Exception** - Exception name | Description - |---|--- - `PulsarAdminException` | Unexpected error + Exception name | Description + |---|--- + `PulsarAdminException` | Unexpected error - For more information, see [`getSinkStatusWithInstanceID`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`getSinkStatusWithInstanceID`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#getSinkStatus-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1178,6 +1194,7 @@ Update a running Pulsar source connector. Use the `update` subcommand. ``` + $ pulsar-admin sources update options ``` @@ -1194,62 +1211,62 @@ Send a `PUT` request to this endpoint: {@inject: endpoint|PUT|/admin/v3/sources/ * Update a running source connector with a **local file**. - ```java - - void updateSource(SourceConfig sourceConfig, - String fileName) - throws PulsarAdminException - - ``` + ```java + + void updateSource(SourceConfig sourceConfig, + String fileName) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - |`sourceConfig` | The source configuration object + | Name | Description + |---|--- + |`sourceConfig` | The source configuration object - **Exception** + **Exception** - |Name|Description| - |---|--- - |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission - | `PulsarAdminException.NotFoundException` | Cluster doesn't exist - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error - For more information, see [`updateSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#updateSource-SourceConfig-java.lang.String-). + For more information, see [`updateSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#updateSource-SourceConfig-java.lang.String-). * Update a source connector using a **remote file** with a URL from which fun-pkg can be downloaded. - ```java - - void updateSourceWithUrl(SourceConfig sourceConfig, - String pkgUrl) - throws PulsarAdminException - - ``` + ```java + + void updateSourceWithUrl(SourceConfig sourceConfig, + String pkgUrl) + throws PulsarAdminException + + ``` - Supported URLs are `http` and `file`. + Supported URLs are `http` and `file`. - **Example** + **Example** - * HTTP: http://www.repo.com/fileName.jar + * HTTP: http://www.repo.com/fileName.jar - * File: file:///dir/fileName.jar + * File: file:///dir/fileName.jar - **Parameter** + **Parameter** - | Name | Description - |---|--- - | `sourceConfig` | The source configuration object - | `pkgUrl` | URL from which pkg can be downloaded + | Name | Description + |---|--- + | `sourceConfig` | The source configuration object + | `pkgUrl` | URL from which pkg can be downloaded - **Exception** + **Exception** - |Name|Description| - |---|--- - |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission - | `PulsarAdminException.NotFoundException` | Cluster doesn't exist - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error For more information, see [`createSourceWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#updateSourceWithUrl-SourceConfig-java.lang.String-). @@ -1283,6 +1300,7 @@ Update a running Pulsar sink connector. Use the `update` subcommand. ``` + $ pulsar-admin sinks update options ``` @@ -1299,62 +1317,62 @@ Send a `PUT` request to this endpoint: {@inject: endpoint|PUT|/admin/v3/sinks/:t * Update a running sink connector with a **local file**. - ```java - - void updateSink(SinkConfig sinkConfig, - String fileName) - throws PulsarAdminException - - ``` + ```java + + void updateSink(SinkConfig sinkConfig, + String fileName) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - |`sinkConfig` | The sink configuration object + | Name | Description + |---|--- + |`sinkConfig` | The sink configuration object - **Exception** + **Exception** - |Name|Description| - |---|--- - |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission - | `PulsarAdminException.NotFoundException` | Cluster doesn't exist - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + | `PulsarAdminException.NotFoundException` | Cluster doesn't exist + | `PulsarAdminException` | Unexpected error - For more information, see [`updateSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#updateSink-SinkConfig-java.lang.String-). + For more information, see [`updateSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#updateSink-SinkConfig-java.lang.String-). * Update a sink connector using a **remote file** with a URL from which fun-pkg can be downloaded. - ```java - - void updateSinkWithUrl(SinkConfig sinkConfig, - String pkgUrl) - throws PulsarAdminException - - ``` + ```java + + void updateSinkWithUrl(SinkConfig sinkConfig, + String pkgUrl) + throws PulsarAdminException + + ``` - Supported URLs are `http` and `file`. + Supported URLs are `http` and `file`. - **Example** + **Example** - * HTTP: http://www.repo.com/fileName.jar + * HTTP: http://www.repo.com/fileName.jar - * File: file:///dir/fileName.jar + * File: file:///dir/fileName.jar - **Parameter** + **Parameter** - | Name | Description - |---|--- - | `sinkConfig` | The sink configuration object - | `pkgUrl` | URL from which pkg can be downloaded + | Name | Description + |---|--- + | `sinkConfig` | The sink configuration object + | `pkgUrl` | URL from which pkg can be downloaded - **Exception** + **Exception** - |Name|Description| - |---|--- - |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission - |`PulsarAdminException.NotFoundException` | Cluster doesn't exist - |`PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + |`PulsarAdminException.NotAuthorizedException`| You don't have the admin permission + |`PulsarAdminException.NotFoundException` | Cluster doesn't exist + |`PulsarAdminException` | Unexpected error For more information, see [`updateSinkWithUrl`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#updateSinkWithUrl-SinkConfig-java.lang.String-). @@ -1394,6 +1412,7 @@ Stop a source connector. Use the `stop` subcommand. ``` + $ pulsar-admin sources stop options ``` @@ -1416,59 +1435,59 @@ For more information, see [here](io-cli.md#stop). * Stop **all** source connectors. - ```java - - void stopSource(String tenant, - String namespace, - String source) - throws PulsarAdminException - - ``` + ```java + + void stopSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-). * Stop a **specified** source connector. - ```java - - void stopSource(String tenant, - String namespace, - String source, - int instanceId) - throws PulsarAdminException - - ``` + ```java + + void stopSource(String tenant, + String namespace, + String source, + int instanceId) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name - `instanceId` | Source instanceID + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`stopSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#stopSource-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1500,6 +1519,7 @@ Stop a sink connector. Use the `stop` subcommand. ``` + $ pulsar-admin sinks stop options ``` @@ -1522,59 +1542,59 @@ For more information, see [here](io-cli.md#stop-1). * Stop **all** sink connectors. - ```java - - void stopSink(String tenant, - String namespace, - String sink) - throws PulsarAdminException - - ``` + ```java + + void stopSink(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-). * Stop a **specified** sink connector. - ```java - - void stopSink(String tenant, - String namespace, - String sink, - int instanceId) - throws PulsarAdminException - - ``` + ```java + + void stopSink(String tenant, + String namespace, + String sink, + int instanceId) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name - `instanceId` | Source instanceID + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`stopSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#stopSink-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1612,6 +1632,7 @@ Restart a source connector. Use the `restart` subcommand. ``` + $ pulsar-admin sources restart options ``` @@ -1634,59 +1655,59 @@ For more information, see [here](io-cli.md#restart). * Restart **all** source connectors. - ```java - - void restartSource(String tenant, - String namespace, - String source) - throws PulsarAdminException - - ``` + ```java + + void restartSource(String tenant, + String namespace, + String source) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-). * Restart a **specified** source connector. - ```java - - void restartSource(String tenant, - String namespace, - String source, - int instanceId) - throws PulsarAdminException - - ``` + ```java + + void restartSource(String tenant, + String namespace, + String source, + int instanceId) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name - `instanceId` | Source instanceID + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Source instanceID - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`restartSource`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Source.html#restartSource-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1718,6 +1739,7 @@ Restart a sink connector. Use the `restart` subcommand. ``` + $ pulsar-admin sinks restart options ``` @@ -1740,59 +1762,59 @@ For more information, see [here](io-cli.md#restart-1). * Restart all Pulsar sink connectors. - ```java - - void restartSink(String tenant, - String namespace, - String sink) - throws PulsarAdminException - - ``` + ```java + + void restartSink(String tenant, + String namespace, + String sink) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `sink` | Sink name + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `sink` | Sink name - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-). + For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-). * Restart a **specified** sink connector. - ```java - - void restartSink(String tenant, - String namespace, - String sink, - int instanceId) - throws PulsarAdminException - - ``` + ```java + + void restartSink(String tenant, + String namespace, + String sink, + int instanceId) + throws PulsarAdminException + + ``` - **Parameter** + **Parameter** - | Name | Description - |---|--- - `tenant` | Tenant name - `namespace` | Namespace name - `source` | Source name - `instanceId` | Sink instanceID + | Name | Description + |---|--- + `tenant` | Tenant name + `namespace` | Namespace name + `source` | Source name + `instanceId` | Sink instanceID - **Exception** + **Exception** - |Name|Description| - |---|--- - | `PulsarAdminException` | Unexpected error + |Name|Description| + |---|--- + | `PulsarAdminException` | Unexpected error - For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-int-). + For more information, see [`restartSink`](https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#restartSink-java.lang.String-java.lang.String-java.lang.String-int-). @@ -1830,6 +1852,7 @@ Delete a source connector. Use the `delete` subcommand. ``` + $ pulsar-admin sources delete options ``` @@ -1906,6 +1929,7 @@ Delete a sink connector. Use the `delete` subcommand. ``` + $ pulsar-admin sinks delete options ``` From 9be7cb8126fdbb983a6420dba83f3a52b9b01458 Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Tue, 26 Oct 2021 15:07:31 +0800 Subject: [PATCH 5/8] patch --- .../versioned_docs/version-2.7.2/schema-get-started.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md index 3cc345855ea4f..c8193ab572c69 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md +++ b/site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md @@ -72,10 +72,6 @@ If you construct a producer without specifying a schema, then the producer can o **Example** ``` -<<<<<<< HEAD -======= - ->>>>>>> up/master Producer producer = client.newProducer() .topic(topic) .create(); @@ -84,10 +80,6 @@ byte[] message = … // serialize the `user` by yourself; producer.send(message); ``` -<<<<<<< HEAD -======= - ->>>>>>> up/master ### With schema If you construct a producer with specifying a schema, then you can send a class to a topic directly without worrying about how to serialize POJOs into bytes. From 087e6963956ea6836f1deaf2779dc57843868bdd Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Wed, 27 Oct 2021 17:37:12 +0800 Subject: [PATCH 6/8] commit 2.7.2 io quick start --- site2/website-next/docs/io-quickstart.md | 5 +---- .../versioned_docs/version-2.7.2/io-quickstart.md | 5 ++--- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/site2/website-next/docs/io-quickstart.md b/site2/website-next/docs/io-quickstart.md index 7f1efb6d54dfc..87356f5fd551e 100644 --- a/site2/website-next/docs/io-quickstart.md +++ b/site2/website-next/docs/io-quickstart.md @@ -453,13 +453,10 @@ This section demonstrates how to connect Pulsar to PostgreSQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics +* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). ::: -and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. ->For more information, see [JDBC sink connector](io-jdbc-sink). - ### Setup a PostgreSQL cluster diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md index 67ca9710a10a8..84421a5a57970 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-quickstart.md @@ -454,12 +454,11 @@ This section demonstrates how to connect Pulsar to PostgreSQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics +* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). ::: -and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. ->For more information, see [JDBC sink connector](io-jdbc-sink). + ### Setup a PostgreSQL cluster From 530a24f3aa400971b60e25eeff0965a8fc6dc5e8 Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Wed, 27 Oct 2021 20:05:37 +0800 Subject: [PATCH 7/8] fix 2.7.3 and 2.8.0 and next --- site2/website-next/docs/admin-api-topics.md | 2 +- .../docs/client-libraries-java.md | 7 +- site2/website-next/docs/io-influxdb-sink.md | 7 + site2/website-next/docs/io-quickstart.md | 15 +- .../website-next/docs/reference-cli-tools.md | 8 + .../docs/reference-configuration.md | 2 + site2/website-next/docs/reference-metrics.md | 2 +- .../website-next/docs/security-encryption.md | 281 ++++++++++++++---- .../docs/sql-deployment-configurations.md | 3 + .../version-2.7.3/io-influxdb-sink.md | 7 + .../version-2.7.3/reference-metrics.md | 1 - .../version-2.8.0/io-influxdb-sink.md | 6 +- 12 files changed, 265 insertions(+), 76 deletions(-) diff --git a/site2/website-next/docs/admin-api-topics.md b/site2/website-next/docs/admin-api-topics.md index ffd020257a2f3..1e5adcd33ee08 100644 --- a/site2/website-next/docs/admin-api-topics.md +++ b/site2/website-next/docs/admin-api-topics.md @@ -1215,7 +1215,7 @@ $ pulsar-admin topics reset-cursor \ String topic = "persistent://my-tenant/my-namespace/my-topic"; String subName = "my-subscription"; long timestamp = 2342343L; -admin.topics().skipAllMessages(topic, subName, timestamp); +admin.topics().resetCursor(topic, subName, timestamp); ``` diff --git a/site2/website-next/docs/client-libraries-java.md b/site2/website-next/docs/client-libraries-java.md index dc54b794baf13..a47c27f06e29d 100644 --- a/site2/website-next/docs/client-libraries-java.md +++ b/site2/website-next/docs/client-libraries-java.md @@ -152,21 +152,24 @@ Check out the Javadoc for the {@inject: javadoc:PulsarClient:/client/org/apache/ > In addition to client-level configuration, you can also apply [producer](#configure-producer) and [consumer](#configure-consumer) specific configuration as described in sections below. ### Client memory allocator configuration -You can set the client memory allocator configurations through Java properties.
    +You can set the client memory allocator configurations through Java properties.
    | Property | Type |
    Description
    | Default | Available values |---|---|---|---|--- -`pulsar.allocator.pooled` | String | If set to `true`, the client uses a direct memory pool.
    If set to `false`, the client uses a heap memory without pool | true |
  • true
  • false
    • +`pulsar.allocator.pooled` | String | If set to `true`, the client uses a direct memory pool.
    If set to `false`, the client uses a heap memory without pool | true |
  • true
  • false
    • `pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM when OOM happens | false |
  • true
  • false
    • `pulsar.allocator.leak_detection` | String | Service URL provider for Pulsar service | Disabled |
  • Disabled
  • Simple
  • Advanced
  • Paranoid
    • `pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the client throws an exception or fallbacks to heap | FallbackToHeap |
  • ThrowException
  • FallbackToHeap
    • **Example**: + ``` + -Dpulsar.allocator.pooled=true -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.leak_detection=Disabled -Dpulsar.allocator.out_of_memory_policy=ThrowException + ``` ## Producer diff --git a/site2/website-next/docs/io-influxdb-sink.md b/site2/website-next/docs/io-influxdb-sink.md index f6912ed0478ef..647ee5d7043fe 100644 --- a/site2/website-next/docs/io-influxdb-sink.md +++ b/site2/website-next/docs/io-influxdb-sink.md @@ -68,7 +68,9 @@ Before using the InfluxDB sink connector, you need to create a configuration fil * YAML + ```yaml + configs: influxdbUrl: "http://localhost:9999" organization: "example-org" @@ -79,7 +81,9 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 + ``` + #### InfluxDBv1 @@ -103,6 +107,7 @@ Before using the InfluxDB sink connector, you need to create a configuration fil * YAML ```yaml + configs: influxdbUrl: "http://localhost:8086" database: "test_db" @@ -112,4 +117,6 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 + ``` + diff --git a/site2/website-next/docs/io-quickstart.md b/site2/website-next/docs/io-quickstart.md index 87356f5fd551e..5215afc59466a 100644 --- a/site2/website-next/docs/io-quickstart.md +++ b/site2/website-next/docs/io-quickstart.md @@ -453,10 +453,13 @@ This section demonstrates how to connect Pulsar to PostgreSQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). +* The JDBC sink connector pulls messages from Pulsar topics ::: +and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. +>For more information, see [JDBC sink connector](io-jdbc-sink). + ### Setup a PostgreSQL cluster @@ -655,7 +658,7 @@ The sink has been created successfully if the following message appears. ```bash -"Created successfully" +Created successfully ``` @@ -803,7 +806,7 @@ The sink instance has been stopped successfully if the following message disappe ```bash -"Stopped successfully" +Stopped successfully ``` @@ -831,7 +834,7 @@ The sink instance has been started successfully if the following message disappe ```bash -"Started successfully" +Started successfully ``` @@ -868,7 +871,7 @@ The sink connector has been updated successfully if the following message disapp ```bash -"Updated successfully" +Updated successfully ``` @@ -937,7 +940,7 @@ The sink connector has been deleted successfully if the following message appear ```text -"Deleted successfully" +Deleted successfully ``` diff --git a/site2/website-next/docs/reference-cli-tools.md b/site2/website-next/docs/reference-cli-tools.md index af1cd0ff1f645..5792cc6080916 100644 --- a/site2/website-next/docs/reference-cli-tools.md +++ b/site2/website-next/docs/reference-cli-tools.md @@ -442,6 +442,7 @@ $ pulsar-daemon command Commands * `start` * `stop` +* `restart` ### `start` @@ -472,7 +473,14 @@ Options |---|---|---| |-force|Stop the service forcefully if not stopped by normal shutdown.|false| +### `restart` +Restart a service that has already been started. +```bash + +$ pulsar-daemon restart service + +``` ## `pulsar-perf` A tool for performance testing a Pulsar broker. diff --git a/site2/website-next/docs/reference-configuration.md b/site2/website-next/docs/reference-configuration.md index 53c4cbfd6d064..a064609d627b8 100644 --- a/site2/website-next/docs/reference-configuration.md +++ b/site2/website-next/docs/reference-configuration.md @@ -483,6 +483,8 @@ You can set the log level and configuration in the [log4j2.yaml](https://github |subscribeRatePeriodPerConsumerInSecond|Rate period for {subscribeThrottlingRatePerConsumer}. By default, it is 30s.|30| | dispatchThrottlingRatePerTopicInMsg | Default messages (per second) dispatch throttling-limit for every topic. When the value is set to 0, default message dispatch throttling-limit is disabled. |0 | | dispatchThrottlingRatePerTopicInByte | Default byte (per second) dispatch throttling-limit for every topic. When the value is set to 0, default byte dispatch throttling-limit is disabled. | 0| +| dispatchThrottlingOnBatchMessageEnabled |Apply dispatch rate limiting on batch message instead individual messages with in batch message. (Default is disabled). | false| + | dispatchThrottlingRateRelativeToPublishRate | Enable dispatch rate-limiting relative to publish rate. | false | |dispatchThrottlingRatePerSubscriptionInMsg|The defaulted number of message dispatching throttling-limit for a subscription. The value of 0 disables message dispatch-throttling.|0| |dispatchThrottlingRatePerSubscriptionInByte|The default number of message-bytes dispatching throttling-limit for a subscription. The value of 0 disables message-byte dispatch-throttling.|0| diff --git a/site2/website-next/docs/reference-metrics.md b/site2/website-next/docs/reference-metrics.md index cd4609f623d37..854b5153af1b5 100644 --- a/site2/website-next/docs/reference-metrics.md +++ b/site2/website-next/docs/reference-metrics.md @@ -311,7 +311,7 @@ All the loadbalancing metrics are labelled with the following labels: | pulsar_lb_bandwidth_in_usage | Gauge | The broker inbound bandwith usage (in percent). | | pulsar_lb_bandwidth_out_usage | Gauge | The broker outbound bandwith usage (in percent). | | pulsar_lb_cpu_usage | Gauge | The broker cpu usage (in percent). | -| pulsar_lb_directMemory_usage | Gauge | The broker process direct memory usage (in percent). | +| pulsar_lb_directMemory_usage | Gauge | The broker process direct memory usage (in percent). | | pulsar_lb_memory_usage | Gauge | The broker process memory usage (in percent). | #### BundleUnloading metrics diff --git a/site2/website-next/docs/security-encryption.md b/site2/website-next/docs/security-encryption.md index 04c32c2233515..419e6d79ce9de 100644 --- a/site2/website-next/docs/security-encryption.md +++ b/site2/website-next/docs/security-encryption.md @@ -45,65 +45,202 @@ openssl ec -in test_ecdsa_privkey.pem -pubout -outform pem -out test_ecdsa_pubke 4. Add encryption key name to producer builder: PulsarClient.newProducer().addEncryptionKey("myapp.key"). -5. Add CryptoKeyReader implementation to producer or consumer builder: PulsarClient.newProducer().cryptoKeyReader(keyReader) / PulsarClient.newConsumer().cryptoKeyReader(keyReader). - -6. Sample producer application: +5. Configure a `CryptoKeyReader` to a producer, consumer or reader. + + + ```java -class RawFileKeyReader implements CryptoKeyReader { +PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(); +String topic = "persistent://my-tenant/my-ns/my-topic"; +// RawFileKeyReader is just an example implementation that's not provided by Pulsar +CryptoKeyReader keyReader = new RawFileKeyReader("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem"); + +Producer producer = pulsarClient.newProducer() + .topic(topic) + .cryptoKeyReader(keyReader) + .addEncryptionKey(“myappkey”) + .create(); + +Consumer consumer = pulsarClient.newConsumer() + .topic(topic) + .subscriptionName("my-subscriber-name") + .cryptoKeyReader(keyReader) + .subscribe(); + +Reader reader = pulsarClient.newReader() + .topic(topic) + .startMessageId(MessageId.earliest) + .cryptoKeyReader(keyReader) + .create(); - String publicKeyFile = ""; - String privateKeyFile = ""; +``` - RawFileKeyReader(String pubKeyFile, String privKeyFile) { - publicKeyFile = pubKeyFile; - privateKeyFile = privKeyFile; - } + + - @Override - public EncryptionKeyInfo getPublicKey(String keyName, Map keyMeta) { - EncryptionKeyInfo keyInfo = new EncryptionKeyInfo(); - try { - keyInfo.setKey(Files.readAllBytes(Paths.get(publicKeyFile))); - } catch (IOException e) { - System.out.println("ERROR: Failed to read public key from file " + publicKeyFile); - e.printStackTrace(); - } - return keyInfo; - } +```c++ - @Override - public EncryptionKeyInfo getPrivateKey(String keyName, Map keyMeta) { - EncryptionKeyInfo keyInfo = new EncryptionKeyInfo(); - try { - keyInfo.setKey(Files.readAllBytes(Paths.get(privateKeyFile))); - } catch (IOException e) { - System.out.println("ERROR: Failed to read private key from file " + privateKeyFile); - e.printStackTrace(); - } - return keyInfo; - } -} +Client client("pulsar://localhost:6650"); +std::string topic = "persistent://my-tenant/my-ns/my-topic"; +// DefaultCryptoKeyReader is a built-in implementation that reads public key and private key from files +auto keyReader = std::make_shared("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem"); -PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(); +Producer producer; +ProducerConfiguration producerConf; +producerConf.setCryptoKeyReader(keyReader); +producerConf.addEncryptionKey("myappkey"); +client.createProducer(topic, producerConf, producer); + +Consumer consumer; +ConsumerConfiguration consumerConf; +consumerConf.setCryptoKeyReader(keyReader); +client.subscribe(topic, "my-subscriber-name", consumerConf, consumer); + +Reader reader; +ReaderConfiguration readerConf; +readerConf.setCryptoKeyReader(keyReader); +client.createReader(topic, MessageId::earliest(), readerConf, reader); + +``` + + + + +```python + +from pulsar import Client, CryptoKeyReader -Producer producer = pulsarClient.newProducer() - .topic("persistent://my-tenant/my-ns/my-topic") - .addEncryptionKey("myappkey") - .cryptoKeyReader(new RawFileKeyReader("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem")) - .create(); +client = Client('pulsar://localhost:6650') +topic = 'persistent://my-tenant/my-ns/my-topic' +# CryptoKeyReader is a built-in implementation that reads public key and private key from files +key_reader = CryptoKeyReader('test_ecdsa_pubkey.pem', 'test_ecdsa_privkey.pem') + +producer = client.create_producer( + topic=topic, + encryption_key='myappkey', + crypto_key_reader=key_reader +) + +consumer = client.subscribe( + topic=topic, + subscription_name='my-subscriber-name', + crypto_key_reader=key_reader +) + +reader = client.create_reader( + topic=topic, + start_message_id=MessageId.earliest, + crypto_key_reader=key_reader +) + +client.close() + +``` + + + + +```nodejs + +const Pulsar = require('pulsar-client'); + +(async () => { +// Create a client +const client = new Pulsar.Client({ + serviceUrl: 'pulsar://localhost:6650', + operationTimeoutSeconds: 30, +}); + +// Create a producer +const producer = await client.createProducer({ + topic: 'persistent://public/default/my-topic', + sendTimeoutMs: 30000, + batchingEnabled: true, + publicKeyPath: "public-key.client-rsa.pem", + encryptionKey: "encryption-key" +}); + +// Create a consumer +const consumer = await client.subscribe({ + topic: 'persistent://public/default/my-topic', + subscription: 'sub1', + subscriptionType: 'Shared', + ackTimeoutMs: 10000, + privateKeyPath: "private-key.client-rsa.pem" +}); + +// Send messages +for (let i = 0; i < 10; i += 1) { + const msg = `my-message-${i}`; + producer.send({ + data: Buffer.from(msg), + }); + console.log(`Sent message: ${msg}`); +} +await producer.flush(); -for (int i = 0; i < 10; i++) { - producer.send("my-message".getBytes()); +// Receive messages +for (let i = 0; i < 10; i += 1) { + const msg = await consumer.receive(); + console.log(msg.getData().toString()); + consumer.acknowledge(msg); } -producer.close(); -pulsarClient.close(); +await consumer.close(); +await producer.close(); +await client.close(); +})(); ``` -7. Sample Consumer Application: + + + + +6. Below is an example of a **customized** `CryptoKeyReader` implementation. + + + ```java @@ -142,27 +279,49 @@ class RawFileKeyReader implements CryptoKeyReader { } } -PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(); -Consumer consumer = pulsarClient.newConsumer() - .topic("persistent://my-tenant/my-ns/my-topic") - .subscriptionName("my-subscriber-name") - .cryptoKeyReader(new RawFileKeyReader("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem")) - .subscribe(); -Message msg = null; - -for (int i = 0; i < 10; i++) { - msg = consumer.receive(); - // do something - System.out.println("Received: " + new String(msg.getData())); -} +``` + + + + +```c++ + +class CustomCryptoKeyReader : public CryptoKeyReader { + public: + Result getPublicKey(const std::string& keyName, std::map& metadata, + EncryptionKeyInfo& encKeyInfo) const override { + // TODO: + return ResultOk; + } + + Result getPrivateKey(const std::string& keyName, std::map& metadata, + EncryptionKeyInfo& encKeyInfo) const override { + // TODO: + return ResultOk; + } +}; -// Acknowledge the consumption of all messages at once -consumer.acknowledgeCumulative(msg); -consumer.close(); -pulsarClient.close(); +auto keyReader = std::make_shared(/* ... */); +// TODO: create producer, consumer or reader based on keyReader here ``` +Besides, you can use the **default** implementation of `CryptoKeyReader` by specifying the paths of `private key` and `public key`. + + + + +Currently, **customized** `CryptoKeyReader` implementation is not supported in Python. However, you can use the **default** implementation by specifying the path of `private key` and `public key`. + + + + +Currently, **customized** `CryptoKeyReader` implementation is not supported in Node.JS. However, you can use the **default** implementation by specifying the path of `private key` and `public key`. + + + + + ## Key rotation Pulsar generates a new AES data key every 4 hours or after publishing a certain number of messages. A producer fetches the asymmetric public key every 4 hours by calling CryptoKeyReader.getPublicKey() to retrieve the latest version. diff --git a/site2/website-next/docs/sql-deployment-configurations.md b/site2/website-next/docs/sql-deployment-configurations.md index 0a74c33f1a454..48f99e4d7a296 100644 --- a/site2/website-next/docs/sql-deployment-configurations.md +++ b/site2/website-next/docs/sql-deployment-configurations.md @@ -30,6 +30,9 @@ pulsar.entry-read-batch-size=100 # default number of splits to use per query pulsar.target-num-splits=4 +# max size of one batch message (default value is 5MB) +pulsar.max-message-size=5242880 + ``` You can connect Presto to a Pulsar cluster with multiple hosts. To configure multiple hosts for brokers, add multiple URLs to `pulsar.web-service-url`. To configure multiple hosts for ZooKeeper, add multiple URIs to `pulsar.zookeeper-uri`. The following is an example. diff --git a/site2/website-next/versioned_docs/version-2.7.3/io-influxdb-sink.md b/site2/website-next/versioned_docs/version-2.7.3/io-influxdb-sink.md index d2dbeedd47895..4023423964e34 100644 --- a/site2/website-next/versioned_docs/version-2.7.3/io-influxdb-sink.md +++ b/site2/website-next/versioned_docs/version-2.7.3/io-influxdb-sink.md @@ -69,7 +69,9 @@ Before using the InfluxDB sink connector, you need to create a configuration fil * YAML + ```yaml + configs: influxdbUrl: "http://localhost:9999" organization: "example-org" @@ -80,7 +82,9 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 + ``` + #### InfluxDBv1 @@ -104,6 +108,7 @@ Before using the InfluxDB sink connector, you need to create a configuration fil * YAML ```yaml + configs: influxdbUrl: "http://localhost:8086" database: "test_db" @@ -113,4 +118,6 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 + ``` + diff --git a/site2/website-next/versioned_docs/version-2.7.3/reference-metrics.md b/site2/website-next/versioned_docs/version-2.7.3/reference-metrics.md index 89c327534c8a3..3e2326a3e72f3 100644 --- a/site2/website-next/versioned_docs/version-2.7.3/reference-metrics.md +++ b/site2/website-next/versioned_docs/version-2.7.3/reference-metrics.md @@ -25,7 +25,6 @@ The following types of metrics are available: - [Counter](https://prometheus.io/docs/concepts/metric_types/#counter): a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster. - [Gauge](https://prometheus.io/docs/concepts/metric_types/#gauge): a metric that represents a single numerical value that can arbitrarily go up and down. - [Histogram](https://prometheus.io/docs/concepts/metric_types/#histogram): a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. The `_bucket` suffix is the number of observations within a histogram bucket, configured with parameter `{le=""}`. The `_count` suffix is the number of observations, shown as a time series and behaves like a counter. The `_sum` suffix is the sum of observed values, also shown as a time series and behaves like a counter. These suffixes are together denoted by `_*` in this doc. - - [Summary](https://prometheus.io/docs/concepts/metric_types/#summary): similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window. ## ZooKeeper diff --git a/site2/website-next/versioned_docs/version-2.8.0/io-influxdb-sink.md b/site2/website-next/versioned_docs/version-2.8.0/io-influxdb-sink.md index 0a0593ee3eaba..4023423964e34 100644 --- a/site2/website-next/versioned_docs/version-2.8.0/io-influxdb-sink.md +++ b/site2/website-next/versioned_docs/version-2.8.0/io-influxdb-sink.md @@ -72,7 +72,7 @@ Before using the InfluxDB sink connector, you need to create a configuration fil ```yaml - { + configs: influxdbUrl: "http://localhost:9999" organization: "example-org" bucket: "example-bucket" @@ -82,7 +82,6 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 - } ``` @@ -110,7 +109,7 @@ Before using the InfluxDB sink connector, you need to create a configuration fil ```yaml - { + configs: influxdbUrl: "http://localhost:8086" database: "test_db" consistencyLevel: "ONE" @@ -119,7 +118,6 @@ Before using the InfluxDB sink connector, you need to create a configuration fil gzipEnable: false batchTimeMs: 1000 batchSize: 100 - } ``` From b9904c385c4ac50c5656e1f7bcf1fb4884c1396a Mon Sep 17 00:00:00 2001 From: Yan Zhang Date: Thu, 28 Oct 2021 14:31:54 +0800 Subject: [PATCH 8/8] fix develop tip --- .../website-next/versioned_docs/version-2.7.2/io-develop.md | 6 +++--- .../website-next/versioned_docs/version-2.7.3/io-develop.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/site2/website-next/versioned_docs/version-2.7.2/io-develop.md b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md index d7531a0d30d67..cbcb555b9b9cc 100644 --- a/site2/website-next/versioned_docs/version-2.7.2/io-develop.md +++ b/site2/website-next/versioned_docs/version-2.7.2/io-develop.md @@ -94,11 +94,11 @@ interface, which means you need to implement the {@inject: github:open:/pulsar-i `ack` |Acknowledge that the record is fully processed. `fail`|Indicate that the record fails to be processed. -:::tip + :::tip -For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. + For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. -::: + ::: ### Sink diff --git a/site2/website-next/versioned_docs/version-2.7.3/io-develop.md b/site2/website-next/versioned_docs/version-2.7.3/io-develop.md index d7531a0d30d67..cbcb555b9b9cc 100644 --- a/site2/website-next/versioned_docs/version-2.7.3/io-develop.md +++ b/site2/website-next/versioned_docs/version-2.7.3/io-develop.md @@ -94,11 +94,11 @@ interface, which means you need to implement the {@inject: github:open:/pulsar-i `ack` |Acknowledge that the record is fully processed. `fail`|Indicate that the record fails to be processed. -:::tip + :::tip -For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. + For more information about **how to create a source connector**, see {@inject: github:KafkaSource:/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java}. -::: + ::: ### Sink