Skip to content

[feat] [tiered-storage] Add pure S3 provider for the offloader - #15710

Merged
zymap merged 2 commits into
apache:masterfrom
zymap:fix-s3-offloader
May 25, 2022
Merged

[feat] [tiered-storage] Add pure S3 provider for the offloader#15710
zymap merged 2 commits into
apache:masterfrom
zymap:fix-s3-offloader

Conversation

@zymap

@zymap zymap commented May 23, 2022

Copy link
Copy Markdown
Member

Motivation

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR #8985 provides
the Aliyun offload provider, but it has a force limitation of
the S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS. That
is not a limitation on other storage services which compatible
with S3 APIs.
This PR provides a more general offload provider S3 which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

Modifications

  • Add the pure S3 offload provider

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • no-need-doc
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-added
    (Docs have been already added)

@zymap
[improve] [tiered-storage] Add pure S3 provider for the offloader
b8e5e77 
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR apache#8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider
@github-actions github-actions Bot added the doc-required Your PR changes impact docs and you will update later. label May 23, 2022
@zymap zymap added area/tieredstorage type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages type/feature The PR added a new feature or issue requested a new feature and removed type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels May 23, 2022
@zymap zymap changed the title [improve] [tiered-storage] Add pure S3 provider for the offloader [feat] [tiered-storage] Add pure S3 provider for the offloader May 23, 2022
@Technoboy- Technoboy- added this to the 2.11.0 milestone May 23, 2022
hangc0276
hangc0276 requested changes May 24, 2022
View reviewed changes
Comment thread ...main/java/org/apache/bookkeeper/mledger/offload/jcloud/provider/JCloudBlobStoreProvider.java
Comment thread ...n/java/org/apache/bookkeeper/mledger/offload/jcloud/provider/TieredStorageConfiguration.java
@zymap
zymap merged commit 047cb0e into apache:master May 25, 2022
@momo-jun

momo-jun commented May 27, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Doc status update

@zymap will add docs for this new offloader soon.

@momo-jun momo-jun mentioned this pull request May 30, 2022
Merged
1 task
This was referenced May 31, 2022
Open
Open
@Anonymitaet Anonymitaet added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Jun 6, 2022
hangc0276 pushed a commit that referenced this pull request Jun 7, 2022
@zymap @hangc0276
[feat] [tiered-storage] Add pure S3 provider for the offloader (#15710)
72629be 
* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR #8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider

(cherry picked from commit 047cb0e)
@hangc0276 hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Jun 7, 2022
codelipenghui pushed a commit to codelipenghui/incubator-pulsar that referenced this pull request Jun 7, 2022
@zymap @codelipenghui
[feat] [tiered-storage] Add pure S3 provider for the offloader (apach…
d1db33e 
…e#15710)

* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR apache#8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider

(cherry picked from commit 047cb0e)
@eolivelli eolivelli mentioned this pull request Jun 7, 2022
Merged
1 task
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jun 7, 2022
@zymap @nicoloboschi
[feat] [tiered-storage] Add pure S3 provider for the offloader (apach…
6a89682 
…e#15710)

* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR apache#8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider

(cherry picked from commit 047cb0e)
(cherry picked from commit 72629be)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jun 7, 2022
@zymap @nicoloboschi
[feat] [tiered-storage] Add pure S3 provider for the offloader (apach…
f554598 
…e#15710)

* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR apache#8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider

(cherry picked from commit 047cb0e)
(cherry picked from commit d1db33e)
codelipenghui pushed a commit that referenced this pull request Jun 10, 2022
@zymap @codelipenghui
[feat] [tiered-storage] Add pure S3 provider for the offloader (#15710)
b2de043 
* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR #8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider

(cherry picked from commit 047cb0e)
@codelipenghui codelipenghui added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Jun 10, 2022
@zymap zymap mentioned this pull request Aug 23, 2022
Merged
5 tasks
dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022
@zymap qunzhong
[feat] [tiered-storage] Add pure S3 provider for the offloader (apach…
1026646 
…e#15710)

* [improve] [tiered-storage] Add pure S3 provider for the offloader
---

*Motivation*

There have some cloud storages are compatible with S3
APIs, such as aliyun-oss. Some other storages also use
the S3 APIs and want to offload the data into them, but
we only support the AWS or the Aliyun.
The PR apache#8985 provides
the Aliyun offload provider, but it has a force limitation of
the `S3Constants.PROPERTY_S3_VIRTUAL_HOST_BUCKETS`. That
is not a limitation on other storage service which compatible
with S3 APIs.
This PR provides  a more general offload provider `S3` which uses
pure JClouds S3 metadata and allows people to override the
default JClouds properties through system properties.

*Modifications*

- Add the pure S3 offload provider
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hangc0276 hangc0276 hangc0276 approved these changes
@codelipenghui codelipenghui Awaiting requested review from codelipenghui
@eolivelli eolivelli Awaiting requested review from eolivelli
@gaoran10 gaoran10 Awaiting requested review from gaoran10

Assignees

@zymap zymap

Labels

area/tieredstorage cherry-picked/branch-2.8 Archived: 2.8 is end of life cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-complete Your PR changes impact docs and the related docs have been already added. release/2.8.4 release/2.9.3 release/2.10.1 type/feature The PR added a new feature or issue requested a new feature

Projects

None yet

Milestone

2.11.0

Development

Successfully merging this pull request may close these issues.

6 participants

@zymap @momo-jun @hangc0276 @Technoboy- @codelipenghui @Anonymitaet