[Issue 8887][tiered-storage-jcloud] support tiered-storage provider by aliyun OSS by yufan022 · Pull Request #8985 · apache/pulsar

yufan022 · 2020-12-17T07:37:27Z

Motivation

support tiered storage by aliyun oss.

Modifications

Aliyun OSS is compatible with the S3 API.
For security reasons, OSS supports only virtual hosted style access.
So I use S3ApiMetadata to connect OSS, and set jclouds.s3.virtual-host-buckets=true.

documents:
China: https://help.aliyun.com/document_detail/64919.html
Internationale: https://www.alibabacloud.com/help/doc-detail/64919.htm

Verifying this change

test with local

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? more tiered-storage provider
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

yufan022 · 2020-12-18T06:23:54Z

/pulsarbot run-failure-checks

sijie · 2020-12-18T07:20:15Z

@yufan022 great contributions!

@gaoran10 @Renkai can you help review this change?

yufan022 · 2020-12-18T09:50:07Z

/pulsarbot run-failure-checks

gaoran10

Great work!

gaoran10 · 2020-12-18T12:05:59Z

+
+        @Override
+        public BlobStore getBlobStore(TieredStorageConfiguration config) {
+            return OSS_BLOB_STORE_BUILDER.getBlobStore(config);


S3_STANDARD_BLOB_STORE_BUILDER, S3_COMMENT_BLOB_STORE_BUILDER or other names? It seems that oss is a universal name.

AWS_S3 enum is already compatible with S3 API.
if use OSS_BLOB_STORE_BUILDER,OSS_VALIDATION , it will only belongs to Aliyun OSS.
But using S3_API_BLOB_STORE_BUILDER,S3_API_VALIDATION make it universal, both make sense.
What's your suggestion?

gaoran10 · 2020-12-18T12:13:21Z

+    ALIYUN_OSS("aliyun-oss", new AnonymousProviderMetadata(new S3ApiMetadata(), "")) {
+        @Override
+        public void validate(TieredStorageConfiguration config) throws IllegalArgumentException {
+            VALIDATION.validate(config);


Does the serviceEndpoint is a necessary configuration? If necessary it's better to add a configuration validationcheck.

Yep, serviceEndpointis a necessary configuration, I will add a new validator.

gaoran10 · 2020-12-18T12:23:27Z

+
+        @Override
+        public void buildCredentials(TieredStorageConfiguration config) {
+            AWS_CREDENTIAL_BUILDER.buildCredentials(config);


Does the Aliyun security validate is the same as AWS? Do we need to add environment configurations for the Aliyun service? Currently, the AWS security configurations could refer to doc.

Environment variables are reuse for AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY .
You mean we should add new environment variables like ALIYUN_ACCESS_KEY_ID, ALIYUN_SECRET_ACCESS_KEY?

jiazhai · 2020-12-20T12:56:40Z

@yufan022 @gaoran10 Could we add some tests for this part of code change?

yufan022 · 2020-12-20T13:02:06Z

@yufan022 @gaoran10 Could we add some tests for this part of code change?

Sure, Is there any example I can refer to? I couldn't find test case about AWS or AZURE

Renkai · 2020-12-21T02:08:38Z

@yufan022 @gaoran10 Could we add some tests for this part of code change?

@jiazhai
Seems we don't have actually run any test on S3/Azureblob/Google cloud storage, only some mocks, so add a storage vendor will not actually change the mocks.

yufan022 · 2020-12-22T09:16:22Z

/pulsarbot run-failure-checks

sijie · 2020-12-22T18:02:08Z

@Renkai okay. make sense.

sijie · 2020-12-22T18:03:18Z

@jiazhai @gaoran10 can you review the change again?

sijie · 2020-12-24T07:11:58Z

Ping @gaoran10 @jiazhai

yufan022 · 2020-12-29T07:40:25Z

Hi, have we made any progress?

gaoran10 · 2020-12-29T08:56:02Z

Sorry for the late response, LGTM.

jiazhai · 2020-12-29T09:04:57Z

@yufan022 Would you please also add the docs for this aliyun offloader in another PR? Or we add this code, but most of the user will not know how to use this.

Here is some reference: https://github.com/apache/pulsar/blob/master/site2/website/versioned_docs/version-2.7.0/tiered-storage-aws.md

yufan022 · 2020-12-29T09:27:35Z

@jiazhai No problem. Will the code be merged into the latest version(2.8.0)? It seems that the 2.8.0 document has not been created in the master branch.

jiazhai · 2020-12-29T10:01:22Z

@jiazhai No problem. Will the code be merged into the latest version(2.8.0)? It seems that the 2.8.0 document has not been created in the master branch.

Thanks @yufan022, For the latest master, the doc is put at : https://github.com/apache/pulsar/tree/master/site2/docs

yufan022 · 2021-02-24T12:19:05Z

Sorry for late.
docs: #9696

yufan022 · 2021-02-25T09:23:57Z

@codelipenghui Hi, Will the code be merged into 2.7.1 or 2.8.0? Do I need to add documents for 2.7.1?

--- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider

* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider

* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)

…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)

…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e) (cherry picked from commit 72629be)

…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e) (cherry picked from commit d1db33e)

* [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR #8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider (cherry picked from commit 047cb0e)

…e#15710) * [improve] [tiered-storage] Add pure S3 provider for the offloader --- *Motivation* There have some cloud storages are compatible with S3 APIs, such as aliyun-oss. Some other storages also use the S3 APIs and want to offload the data into them, but we only support the AWS or the Aliyun. The PR apache#8985 provides the Aliyun offload provider, but it has a force limitation of the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That is not a limitation on other storage service which compatible with S3 APIs. This PR provides a more general offload provider `S3` which uses pure JClouds S3 metadata and allows people to override the default JClouds properties through system properties. *Modifications* - Add the pure S3 offload provider

support tiered-storage provider by aliyun OSS (apache#8887)

456f5d2

sijie requested review from codelipenghui, jiazhai and merlimat December 18, 2020 07:21

sijie assigned yufan022 Dec 18, 2020

sijie added the area/tieredstorage label Dec 18, 2020

sijie added this to the 2.8.0 milestone Dec 18, 2020

sijie added type/feature The PR added a new feature or issue requested a new feature doc-required Your PR changes impact docs and you will update later. labels Dec 18, 2020

Renkai reviewed Dec 18, 2020

View reviewed changes

Comment thread ...main/java/org/apache/bookkeeper/mledger/offload/jcloud/provider/JCloudBlobStoreProvider.java Outdated

support tiered-storage provider by aliyun OSS (apache#8887)

5f70004

gaoran10 reviewed Dec 18, 2020

View reviewed changes

add new validator by aliyun-oss

2aa5a00

yufan022 force-pushed the issue_8887 branch from 83493cc to 2aa5a00 Compare December 20, 2020 13:30

yufan022 added 2 commits December 21, 2020 12:10

add environment variable by aliyun-oss

cae7456

extract constant

f752890

sijie approved these changes Dec 29, 2020

View reviewed changes

jiazhai approved these changes Dec 29, 2020

View reviewed changes

codelipenghui approved these changes Dec 29, 2020

View reviewed changes

codelipenghui merged commit 98ad39f into apache:master Dec 29, 2020

codelipenghui added release/2.7.1 and removed release/2.7.1 labels Jan 31, 2021

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Apr 12, 2021

zymap mentioned this pull request May 23, 2022

[feat] [tiered-storage] Add pure S3 provider for the offloader #15710

Merged

5 tasks

Uh oh!

Conversation

yufan022 commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Documentation

Uh oh!

yufan022 commented Dec 18, 2020

Uh oh!

sijie commented Dec 18, 2020

Uh oh!

Uh oh!

yufan022 commented Dec 18, 2020

Uh oh!

gaoran10 left a comment

Choose a reason for hiding this comment

Uh oh!

gaoran10 Dec 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yufan022 Dec 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaoran10 Dec 18, 2020

Choose a reason for hiding this comment

Uh oh!

yufan022 Dec 20, 2020

Choose a reason for hiding this comment

Uh oh!

gaoran10 Dec 18, 2020

Choose a reason for hiding this comment

Uh oh!

yufan022 Dec 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiazhai commented Dec 20, 2020

Uh oh!

yufan022 commented Dec 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Renkai commented Dec 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yufan022 commented Dec 22, 2020

Uh oh!

sijie commented Dec 22, 2020

Uh oh!

sijie commented Dec 22, 2020

Uh oh!

sijie commented Dec 24, 2020

Uh oh!

yufan022 commented Dec 29, 2020

Uh oh!

gaoran10 commented Dec 29, 2020

Uh oh!

jiazhai commented Dec 29, 2020

Uh oh!

yufan022 commented Dec 29, 2020

Uh oh!

jiazhai commented Dec 29, 2020

Uh oh!

yufan022 commented Feb 24, 2021

Uh oh!

yufan022 commented Feb 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

yufan022 commented Dec 17, 2020 •

edited

Loading

gaoran10 Dec 18, 2020 •

edited

Loading

yufan022 Dec 20, 2020 •

edited

Loading

yufan022 Dec 20, 2020 •

edited

Loading

yufan022 commented Dec 20, 2020 •

edited

Loading

Renkai commented Dec 21, 2020 •

edited

Loading