From 7d28c6ab80160291e6d0f785ac9b6943e414bb22 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Wed, 24 Feb 2021 20:15:05 +0800 Subject: [PATCH 01/21] [docs] Add documents for Aliyun OSS offload --- site2/docs/tiered-storage-aliyun.md | 243 ++++++++++++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 site2/docs/tiered-storage-aliyun.md diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md new file mode 100644 index 0000000000000..cb34d0c651d6e --- /dev/null +++ b/site2/docs/tiered-storage-aliyun.md @@ -0,0 +1,243 @@ +--- +id: tiered-storage-aliyun +title: Use Aliyun OSS offloader with Pulsar +sidebar_label: Aliyun OSS offloader +--- + +This chapter guides you through every step of installing and configuring the Aliyun Object Storage Service offloader and using it with Pulsar. + +## Installation + +Follow the steps below to install the Aliyun OSS offloader. + +### Prerequisite + +- Pulsar: 2.8.0 or later versions + +### Step + +This example uses Pulsar 2.8.0. + +1. Download the Pulsar tarball using one of the following ways: + + * Download from the [Apache mirror](https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz) + + * Download from the Pulsar [downloads page](https://pulsar.apache.org/download) + + * Use [wget](https://www.gnu.org/software/wget): + + ```shell + wget https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz + ``` + +2. Download and untar the Pulsar offloaders package. + + ```bash + wget https://downloads.apache.org/pulsar/pulsar-2.8.0/apache-pulsar-offloaders-2.8.0-bin.tar.gz + tar xvfz apache-pulsar-offloaders-2.8.0-bin.tar.gz + ``` + +3. Copy the Pulsar offloaders as `offloaders` in the Pulsar directory. + + ``` + mv apache-pulsar-offloaders-2.8.0/offloaders apache-pulsar-2.8.0/offloaders + + ls offloaders + ``` + + **Output** + + As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home) and [Aliyun OSS](https://www.aliyun.com/product/oss) for long term storage. + + + ``` + tiered-storage-file-system-2.8.0.nar + tiered-storage-jcloud-2.8.0.nar + ``` + + > #### Note + > + > * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. + > + > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. + +## Configuration + +> #### Note +> +> Before offloading data from BookKeeper to Aliyun OSS, you need to configure some properties of the Aliyun OSS offload driver. + +Besides, you can also configure the Aliyun OSS offloader to run it automatically or trigger it manually. + +### Configure Aliyun OSS offloader driver + +You can configure the Aliyun OSS offloader driver in the configuration file `broker.conf` or `standalone.conf`. + +- **Required** configurations are as below. + + Required configuration | Description | Example value + |---|---|--- + `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | aliyun-oss + `offloadersDirectory` | Offloader directory | offloaders + `managedLedgerOffloadBucket` | Bucket | pulsar-topic-offload + `managedLedgerOffloadServiceEndpoint` | Endpoint | http://oss-cn-hongkong.aliyuncs.com + +- **Optional** configurations are as below. + + Optional | Description | Example value + |---|---|--- + `managedLedgerOffloadReadBufferSizeInBytes`|Size of block read|1 MB + `managedLedgerOffloadMaxBlockSizeInBytes`|Size of block write|64 MB + `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic

**Note**: it is not recommended that you set this configuration in the production environment.|2 + `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.

**Note**: it is not recommended that you set this configuration in the production environment.|5000 + +#### Bucket (required) + +A bucket is a basic container that holds your data. Everything you store in Aliyun OSS must be contained in a bucket. You can use a bucket to organize your data and control access to your data, but unlike directory and folder, you cannot nest a bucket. + +##### Example + +This example names the bucket as _pulsar-topic-offload_. + +```conf +managedLedgerOffloadBucket=pulsar-topic-offload +``` + +#### Endpoint (required) + +The endpoint is the region where a bucket is located. + +> #### Tip +> +> For more information about Aliyun OSS regions and endpoints, see [here](https://help.aliyun.com/document_detail/31837.html). + +##### Example + +This example sets the endpoint as _oss-us-west-1-internal_. + +``` +managedLedgerOffloadServiceEndpoint=http://oss-us-west-1-internal.aliyuncs.com +``` + +#### Authentication (required) + +To be able to access Aliyun OSS, you need to authenticate with Aliyun OSS. + +* Set the environment variables `ALIYUN_OSS_ACCESS_KEY_ID` and `ALIYUN_OSS_ACCESS_KEY_SECRET` in `conf/pulsar_env.sh`. + + "export" is important so that the variables are made available in the environment of spawned processes. + + ```bash + export ALIYUN_OSS_ACCESS_KEY_ID=ABC123456789 + export ALIYUN_OSS_ACCESS_KEY_SECRET=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c + ``` + +#### Size of block read/write + +You can configure the size of a request sent to or read from Aliyun OSS in the configuration file `broker.conf` or `standalone.conf`. + +Configuration|Description|Default value +|---|---|--- +`managedLedgerOffloadReadBufferSizeInBytes`|Block size for each individual read when reading back data from Aliyun OSS.|1 MB +`managedLedgerOffloadMaxBlockSizeInBytes`|Maximum size of a "part" sent during a multipart upload to Aliyun OSS. It **cannot** be smaller than 5 MB. |64 MB + +### Configure Aliyun OSS offloader to run automatically + +Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically. + +Threshold value|Action +|---|--- +> 0 | It triggers the offloading operation if the topic storage reaches its threshold. += 0|It causes a broker to offload data as soon as possible. +< 0 |It disables automatic offloading operation. + +Automatic offloading runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, offloader does not work until the current segment is full. + +You can configure the threshold size using CLI tools, such as pulsar-admin. + +The offload configurations in `broker.conf` and `standalone.conf` are used for the namespaces that do not have namespace level offload policies. Each namespace can have its own offload policy. If you want to set offload policy for each namespace, use the command [`pulsar-admin namespaces set-offload-policies options`](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-set-offload-policies-em-) command. + +#### Example + +This example sets the Aliyun OSS offloader threshold size to 10 MB using pulsar-admin. + +```bash +bin/pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namespace +``` + +> #### Tip +> +> For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-set-offload-threshold-em-). + +### Configure Aliyun OSS offloader to run manually + +For individual topics, you can trigger Aliyun OSS offloader manually using one of the following methods: + +- Use REST endpoint. + +- Use CLI tools (such as pulsar-admin). + + To trigger it via CLI tools, you need to specify the maximum amount of data (threshold) that should be retained on a Pulsar cluster for a topic. If the size of the topic data on the Pulsar cluster exceeds this threshold, segments from the topic are moved to Aliyun OSS until the threshold is no longer exceeded. Older segments are moved first. + +#### Example + +- This example triggers the Aliyun OSS offloader to run manually using pulsar-admin. + + ```bash + bin/pulsar-admin topics offload --size-threshold 10M my-tenant/my-namespace/topic1 + ``` + + **Output** + + ```bash + Offload triggered for persistent://my-tenant/my-namespace/topic1 for messages before 2:0:-1 + ``` + + > #### Tip + > + > For more information about the `pulsar-admin topics offload options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-offload-em-). + +- This example checks the Aliyun OSS offloader status using pulsar-admin. + + ```bash + bin/pulsar-admin topics offload-status persistent://my-tenant/my-namespace/topic1 + ``` + + **Output** + + ```bash + Offload is currently running + ``` + + To wait for the Aliyun OSS offloader to complete the job, add the `-w` flag. + + ```bash + bin/pulsar-admin topics offload-status -w persistent://my-tenant/my-namespace/topic1 + ``` + + **Output** + + ``` + Offload was a success + ``` + + + If there is an error in offloading, the error is propagated to the `pulsar-admin topics offload-status` command. + + ```bash + bin/pulsar-admin topics offload-status persistent://my-tenant/my-namespace/topic1 + ``` + + **Output** + + ``` + Error in offload + null + + Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads. Please authenticate. (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 798758DE3F1776DF; S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhrtsDlzoOhocuYMpKihQGXe6EG8puRGOkK6UwqzVrMXTWBxxHcS+g=), S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhrtsDlzoOhocuYMpKihQGXe6EG8puRGOkK6UwqzVrMXTWBxxHcS+g= + ```` + + > #### Tip + > + > For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-offload-status-em-). + From 87d08b4ff1e27b0e01935dbe5d316bd295174566 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:09:48 +0800 Subject: [PATCH 02/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index cb34d0c651d6e..38684e8033893 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -4,7 +4,7 @@ title: Use Aliyun OSS offloader with Pulsar sidebar_label: Aliyun OSS offloader --- -This chapter guides you through every step of installing and configuring the Aliyun Object Storage Service offloader and using it with Pulsar. +This chapter guides you through every step of installing and configuring the Aliyun Object Storage Service (OSS) offloader and using it with Pulsar. ## Installation @@ -240,4 +240,3 @@ For individual topics, you can trigger Aliyun OSS offloader manually using one o > #### Tip > > For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-offload-status-em-). - From 444a77d771b06bbd1cf4e4372bf9f21ac9d1d945 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:10:03 +0800 Subject: [PATCH 03/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 38684e8033893..57e567309823d 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -47,7 +47,7 @@ This example uses Pulsar 2.8.0. **Output** - As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home) and [Aliyun OSS](https://www.aliyun.com/product/oss) for long term storage. + As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. ``` From 96977d0558fbe8f87a09bf628abcf0fbec7d0650 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:10:25 +0800 Subject: [PATCH 04/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 57e567309823d..fb1856c8914ba 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -57,7 +57,7 @@ This example uses Pulsar 2.8.0. > #### Note > - > * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. + > * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. > > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. From 6fe83dbdd57ef972b508773979f86cdfe40a0b3b Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:10:34 +0800 Subject: [PATCH 05/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index fb1856c8914ba..39f7c8cc23bcf 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -151,7 +151,7 @@ Threshold value|Action = 0|It causes a broker to offload data as soon as possible. < 0 |It disables automatic offloading operation. -Automatic offloading runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, offloader does not work until the current segment is full. +Automatic offloading runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full. You can configure the threshold size using CLI tools, such as pulsar-admin. From e59684ed3fef90e992f66dbec97573a294f05428 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:10:40 +0800 Subject: [PATCH 06/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 39f7c8cc23bcf..dd1d84fb0406d 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -171,7 +171,7 @@ bin/pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namesp ### Configure Aliyun OSS offloader to run manually -For individual topics, you can trigger Aliyun OSS offloader manually using one of the following methods: +For individual topics, you can trigger the Aliyun OSS offloader manually using one of the following methods: - Use REST endpoint. From b0ec1a561aef6a39f2937250054e7f4fa6b11885 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Thu, 25 Feb 2021 10:18:17 +0800 Subject: [PATCH 07/21] [docs] Add documents for Aliyun OSS offload --- site2/website/sidebars.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/site2/website/sidebars.json b/site2/website/sidebars.json index e21f9b656a22e..03a16fd01aa67 100644 --- a/site2/website/sidebars.json +++ b/site2/website/sidebars.json @@ -55,7 +55,8 @@ "tiered-storage-aws", "tiered-storage-gcs", "tiered-storage-filesystem", - "tiered-storage-azure" + "tiered-storage-azure", + "tiered-storage-aliyun" ], "Transactions": [ "transactions", From 5f556b817cce0511443cd9dfa2275e520be4ebc8 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 17:51:08 +0800 Subject: [PATCH 08/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index dd1d84fb0406d..78f402a7b4cd0 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -47,7 +47,7 @@ This example uses Pulsar 2.8.0. **Output** - As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. + As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. ``` From 8216a3a918092f9a40484318fc151e218fa6b117 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 17:55:00 +0800 Subject: [PATCH 09/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 78f402a7b4cd0..21e8260173ed3 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -169,7 +169,7 @@ bin/pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namesp > > For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-set-offload-threshold-em-). -### Configure Aliyun OSS offloader to run manually +### Run Aliyun OSS offloader manually For individual topics, you can trigger the Aliyun OSS offloader manually using one of the following methods: From 2fd5bbad1acc7012946509841ad6696201df229f Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 17:55:17 +0800 Subject: [PATCH 10/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 21e8260173ed3..4730785a26cf7 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -141,7 +141,7 @@ Configuration|Description|Default value `managedLedgerOffloadReadBufferSizeInBytes`|Block size for each individual read when reading back data from Aliyun OSS.|1 MB `managedLedgerOffloadMaxBlockSizeInBytes`|Maximum size of a "part" sent during a multipart upload to Aliyun OSS. It **cannot** be smaller than 5 MB. |64 MB -### Configure Aliyun OSS offloader to run automatically +### Run Aliyun OSS offloader automatically Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically. From 535075b9bb1ecb0611627d9dccf1375a206c0733 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:10:08 +0800 Subject: [PATCH 11/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 4730785a26cf7..a84dd8a393981 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -55,7 +55,7 @@ This example uses Pulsar 2.8.0. tiered-storage-jcloud-2.8.0.nar ``` - > #### Note + > **Note** > > * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. > From 8e2fdf951762a835f2c723c93d7fd4398c538d5e Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:11:06 +0800 Subject: [PATCH 12/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index a84dd8a393981..04c9f783eb55a 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -59,7 +59,7 @@ This example uses Pulsar 2.8.0. > > * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. > - > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. + > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ## Configuration From 56a0366f1245b76ce25f4d3296406300114a5138 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:11:21 +0800 Subject: [PATCH 13/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 04c9f783eb55a..f35e97420aa4d 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -63,7 +63,7 @@ This example uses Pulsar 2.8.0. ## Configuration -> #### Note +> **Note** > > Before offloading data from BookKeeper to Aliyun OSS, you need to configure some properties of the Aliyun OSS offload driver. From 3284404572a1fde2011dadaa03b947d182b35b14 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:11:39 +0800 Subject: [PATCH 14/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index f35e97420aa4d..6f3fddd529478 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -107,7 +107,7 @@ managedLedgerOffloadBucket=pulsar-topic-offload The endpoint is the region where a bucket is located. -> #### Tip +> **Tip** > > For more information about Aliyun OSS regions and endpoints, see [here](https://help.aliyun.com/document_detail/31837.html). From a4d9750ad60b072ff59411711cfa71794817b049 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:20:37 +0800 Subject: [PATCH 15/21] fix typo --- site2/docs/tiered-storage-aliyun.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index a84dd8a393981..6a46a1a2d4f4b 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -123,14 +123,14 @@ managedLedgerOffloadServiceEndpoint=http://oss-us-west-1-internal.aliyuncs.com To be able to access Aliyun OSS, you need to authenticate with Aliyun OSS. -* Set the environment variables `ALIYUN_OSS_ACCESS_KEY_ID` and `ALIYUN_OSS_ACCESS_KEY_SECRET` in `conf/pulsar_env.sh`. +Set the environment variables `ALIYUN_OSS_ACCESS_KEY_ID` and `ALIYUN_OSS_ACCESS_KEY_SECRET` in `conf/pulsar_env.sh`. - "export" is important so that the variables are made available in the environment of spawned processes. +"export" is important so that the variables are made available in the environment of spawned processes. - ```bash - export ALIYUN_OSS_ACCESS_KEY_ID=ABC123456789 - export ALIYUN_OSS_ACCESS_KEY_SECRET=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c - ``` +```bash +export ALIYUN_OSS_ACCESS_KEY_ID=ABC123456789 +export ALIYUN_OSS_ACCESS_KEY_SECRET=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c +``` #### Size of block read/write From 53aceccd2748e13e82cf5d82fc305a3da4c0d1ee Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 18:26:46 +0800 Subject: [PATCH 16/21] fix typo --- site2/docs/tiered-storage-aliyun.md | 42 ++++++++++++++--------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 6a46a1a2d4f4b..cd4504f56667f 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -75,21 +75,21 @@ You can configure the Aliyun OSS offloader driver in the configuration file `bro - **Required** configurations are as below. - Required configuration | Description | Example value - |---|---|--- - `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | aliyun-oss - `offloadersDirectory` | Offloader directory | offloaders - `managedLedgerOffloadBucket` | Bucket | pulsar-topic-offload - `managedLedgerOffloadServiceEndpoint` | Endpoint | http://oss-cn-hongkong.aliyuncs.com + | Required configuration | Description | Example value | + | --- | --- |--- | + | `managedLedgerOffloadDriver` | Offloader driver name, which is case-insensitive. | aliyun-oss | + | `offloadersDirectory` | Offloader directory | offloaders | + | `managedLedgerOffloadBucket` | Bucket | pulsar-topic-offload | + | `managedLedgerOffloadServiceEndpoint` | Endpoint | http://oss-cn-hongkong.aliyuncs.com | - **Optional** configurations are as below. - Optional | Description | Example value - |---|---|--- - `managedLedgerOffloadReadBufferSizeInBytes`|Size of block read|1 MB - `managedLedgerOffloadMaxBlockSizeInBytes`|Size of block write|64 MB - `managedLedgerMinLedgerRolloverTimeMinutes`|Minimum time between ledger rollover for a topic

**Note**: it is not recommended that you set this configuration in the production environment.|2 - `managedLedgerMaxEntriesPerLedger`|Maximum number of entries to append to a ledger before triggering a rollover.

**Note**: it is not recommended that you set this configuration in the production environment.|5000 + | Optional | Description | Example value | + | --- | --- | --- | + | `managedLedgerOffloadReadBufferSizeInBytes` | Size of block read | 1 MB | + | `managedLedgerOffloadMaxBlockSizeInBytes` | Size of block write | 64 MB | + | `managedLedgerMinLedgerRolloverTimeMinutes` | Minimum time between ledger rollover for a topic

**Note**: it is not recommended that you set this configuration in the production environment. | 2 | + | `managedLedgerMaxEntriesPerLedger` | Maximum number of entries to append to a ledger before triggering a rollover.

**Note**: it is not recommended that you set this configuration in the production environment. | 5000 | #### Bucket (required) @@ -136,20 +136,20 @@ export ALIYUN_OSS_ACCESS_KEY_SECRET=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c You can configure the size of a request sent to or read from Aliyun OSS in the configuration file `broker.conf` or `standalone.conf`. -Configuration|Description|Default value -|---|---|--- -`managedLedgerOffloadReadBufferSizeInBytes`|Block size for each individual read when reading back data from Aliyun OSS.|1 MB -`managedLedgerOffloadMaxBlockSizeInBytes`|Maximum size of a "part" sent during a multipart upload to Aliyun OSS. It **cannot** be smaller than 5 MB. |64 MB +| Configuration | Description | Default value | +| --- | --- | --- | +| `managedLedgerOffloadReadBufferSizeInBytes` | Block size for each individual read when reading back data from Aliyun OSS. | 1 MB | +| `managedLedgerOffloadMaxBlockSizeInBytes` | Maximum size of a "part" sent during a multipart upload to Aliyun OSS. It **cannot** be smaller than 5 MB. | 64 MB | ### Run Aliyun OSS offloader automatically Namespace policy can be configured to offload data automatically once a threshold is reached. The threshold is based on the size of data that a topic has stored on a Pulsar cluster. Once the topic reaches the threshold, an offloading operation is triggered automatically. -Threshold value|Action -|---|--- -> 0 | It triggers the offloading operation if the topic storage reaches its threshold. -= 0|It causes a broker to offload data as soon as possible. -< 0 |It disables automatic offloading operation. +| Threshold value | Action | +| --- | --- | +| > 0 | It triggers the offloading operation if the topic storage reaches its threshold. | +| = 0 | It causes a broker to offload data as soon as possible. | +| < 0 | It disables automatic offloading operation. | Automatic offloading runs when a new segment is added to a topic log. If you set the threshold on a namespace, but few messages are being produced to the topic, the offloader does not work until the current segment is full. From b47ae8e65c64cb31b644853a6d8e7ea7d137148a Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 20:26:00 +0800 Subject: [PATCH 17/21] fix typo --- site2/docs/tiered-storage-aliyun.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index a6cc971e91cfa..1e3822cd817bf 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -109,7 +109,7 @@ The endpoint is the region where a bucket is located. > **Tip** > -> For more information about Aliyun OSS regions and endpoints, see [here](https://help.aliyun.com/document_detail/31837.html). +> For more information about Aliyun OSS regions and endpoints, see [International website](https://www.alibabacloud.com/help/doc-detail/31837.htm) or [Chinese website](https://help.aliyun.com/document_detail/31837.html). ##### Example @@ -221,7 +221,6 @@ For individual topics, you can trigger the Aliyun OSS offloader manually using o Offload was a success ``` - If there is an error in offloading, the error is propagated to the `pulsar-admin topics offload-status` command. ```bash From 434ba171ef2c3cfc90e3e808033aaf17b2f5c541 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 20:26:46 +0800 Subject: [PATCH 18/21] fix typo --- site2/docs/tiered-storage-aliyun.md | 50 ++++++++--------------------- 1 file changed, 13 insertions(+), 37 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 1e3822cd817bf..afd94fc023bcb 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -18,48 +18,24 @@ Follow the steps below to install the Aliyun OSS offloader. This example uses Pulsar 2.8.0. -1. Download the Pulsar tarball using one of the following ways: +1. Download the Pulsar tarball, see [here](https://pulsar.apache.org/docs/en/standalone/#install-pulsar-using-binary-release). - * Download from the [Apache mirror](https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz) +2. Download and untar the Pulsar offloaders package, then copy the Pulsar offloaders as `offloaders` in the Pulsar directory, see [here](https://pulsar.apache.org/docs/en/standalone/#install-tiered-storage-offloaders-optional). - * Download from the Pulsar [downloads page](https://pulsar.apache.org/download) +**Output** - * Use [wget](https://www.gnu.org/software/wget): +As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. - ```shell - wget https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz - ``` - -2. Download and untar the Pulsar offloaders package. - - ```bash - wget https://downloads.apache.org/pulsar/pulsar-2.8.0/apache-pulsar-offloaders-2.8.0-bin.tar.gz - tar xvfz apache-pulsar-offloaders-2.8.0-bin.tar.gz - ``` - -3. Copy the Pulsar offloaders as `offloaders` in the Pulsar directory. - - ``` - mv apache-pulsar-offloaders-2.8.0/offloaders apache-pulsar-2.8.0/offloaders - - ls offloaders - ``` - - **Output** - - As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. - - - ``` - tiered-storage-file-system-2.8.0.nar - tiered-storage-jcloud-2.8.0.nar - ``` +``` +tiered-storage-file-system-2.8.0.nar +tiered-storage-jcloud-2.8.0.nar +``` - > **Note** - > - > * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. - > - > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +> **Note** +> +> * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. +> +> * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ## Configuration From a373185361294a9b7702cbb7ac635e45fd653d6a Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 20:27:39 +0800 Subject: [PATCH 19/21] fix typo --- site2/docs/tiered-storage-aliyun.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index afd94fc023bcb..e76a2a8e42b01 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -22,20 +22,20 @@ This example uses Pulsar 2.8.0. 2. Download and untar the Pulsar offloaders package, then copy the Pulsar offloaders as `offloaders` in the Pulsar directory, see [here](https://pulsar.apache.org/docs/en/standalone/#install-tiered-storage-offloaders-optional). -**Output** - -As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. - -``` -tiered-storage-file-system-2.8.0.nar -tiered-storage-jcloud-2.8.0.nar -``` + **Output** + + As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. + + ``` + tiered-storage-file-system-2.8.0.nar + tiered-storage-jcloud-2.8.0.nar + ``` -> **Note** -> -> * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. -> -> * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. + > **Note** + > + > * If you are running Pulsar in a bare-metal cluster, make sure that `offloaders` tarball is unzipped in every broker's Pulsar directory. + > + > * If you are running Pulsar in Docker or deploying Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ## Configuration From 72287f90613b5c775c7a8a460c870a14a025ac79 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 20:50:38 +0800 Subject: [PATCH 20/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index e76a2a8e42b01..44014c0ce7c36 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -212,6 +212,6 @@ For individual topics, you can trigger the Aliyun OSS offloader manually using o Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads. Please authenticate. (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 798758DE3F1776DF; S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhrtsDlzoOhocuYMpKihQGXe6EG8puRGOkK6UwqzVrMXTWBxxHcS+g=), S3 Extended Request ID: dhBFz/lZm1oiG/oBEepeNlhrtsDlzoOhocuYMpKihQGXe6EG8puRGOkK6UwqzVrMXTWBxxHcS+g= ```` - > #### Tip + > **Tip** > > For more information about the `pulsar-admin topics offload-status options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-offload-status-em-). From a84b7e318ff5a2da1124d1cc5c0e54ca6f1a2189 Mon Sep 17 00:00:00 2001 From: wangyufan Date: Fri, 9 Apr 2021 20:52:05 +0800 Subject: [PATCH 21/21] Update site2/docs/tiered-storage-aliyun.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> --- site2/docs/tiered-storage-aliyun.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/tiered-storage-aliyun.md b/site2/docs/tiered-storage-aliyun.md index 44014c0ce7c36..dab395bb44840 100644 --- a/site2/docs/tiered-storage-aliyun.md +++ b/site2/docs/tiered-storage-aliyun.md @@ -141,7 +141,7 @@ This example sets the Aliyun OSS offloader threshold size to 10 MB using pulsar- bin/pulsar-admin namespaces set-offload-threshold --size 10M my-tenant/my-namespace ``` -> #### Tip +> **Tip** > > For more information about the `pulsar-admin namespaces set-offload-threshold options` command, including flags, descriptions, and default values, see [here](http://pulsar.apache.org/tools/pulsar-admin/2.6.0-SNAPSHOT/#-em-set-offload-threshold-em-).