ARROW-8147: [C++] Add google-cloud-cpp to ThirdpartyToolchain by josiahyan · Pull Request #8757 · apache/arrow

josiahyan · 2020-11-24T14:57:40Z

I'm hoping to work on parts of the GCS filesystem implementation, and this is a prerequisite.

josiahyan · 2020-11-24T15:10:28Z

Running the cmake autoformatter heavily modifies the touched cmake files; should I be doing that?

github-actions · 2020-11-24T15:36:52Z

https://issues.apache.org/jira/browse/ARROW-8147

xhochy · 2020-11-24T19:16:39Z

Running the cmake autoformatter heavily modifies the touched cmake files; should I be doing that?

Maybe you have a different version of that installed and that formats differently? You can also summon the autoformatter with @github-actions<space>autotune on the PR.

josiahyan · 2020-11-25T00:04:49Z

I'll take a look! Didn't know I could do that.

josiahyan · 2020-11-25T01:08:38Z

@github-actions autotune

josiahyan · 2020-11-25T01:32:35Z

@xhochy I can't seem to summon the bot. But I read what the action did, and saw that it required a very specific version of the cmake autoformatter, as you suggested. It works now! Thanks for your help!

xhochy · 2020-11-25T14:37:14Z

@xhochy I can't seem to summon the bot. But I read what the action did, and saw that it required a very specific version of the cmake autoformatter, as you suggested. It works now! Thanks for your help!

The bot did run but didn't make any changes to your CMake files: https://github.com/apache/arrow/runs/1450988053?check_suite_focus=true#step:10:11

josiahyan · 2020-11-25T14:47:30Z

Oh, I know why! The CI queue was full (my test runs were taking hours to go through). By the time the runner got to my PR (Wed, 25 Nov 2020 02:15:47 GMT), my final commit with the manual cmake autoformat had gone in about an hour before.

josiahyan · 2020-12-01T15:48:16Z

@github-actions rebase

nealrichardson · 2021-01-29T17:30:01Z

Hi @josiahyan, apologies for the delay here. Could you please rebase this? Hopefully we can get this reviewed and merged soon after.

josiahyan · 2021-01-29T17:42:41Z

Whoops no problem; I’ll rebase it tonight or over the weekend. Thanks for picking this up again!

…

On Fri, Jan 29, 2021 at 12:30 PM Neal Richardson ***@***.***> wrote: Hi @josiahyan <https://github.com/josiahyan>, apologies for the delay here. Could you please rebase this? Hopefully we can get this reviewed and merged soon after. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8757 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPH2DGRRPB7VG6QSWOYL4DS4LWCVANCNFSM4UA73NKQ> .

josiahyan · 2021-02-28T04:32:16Z

Sorry about the delay! I've rebased the commits and manually merged in the changes made in master.

xhochy · 2021-02-28T07:12:03Z

What is the reason for the exact pin here?

The currently available version in conda (1.24) is incompatible with the way I initially packaged it. I believe I saw in the CI run that storage_client was missing or something, as the google-cloud-cpp package has started namespacing its exports (finally!).

Given the hard link against abseil libraries (as gRPC does too), to reduce packaging size, I decided that the version was safer pinned. The abseil libraries actually loaded probably depend on the matrix of abseil versions and users of it (gRPC, and the proposed google-cloud-cpp inclusion), making it more brittle. Personally, I extract it by a bazel query (provided in comments) and a Python script to lookup the symbols in the ar files. I believe the gRPC builds do something similar, or just pull them out from the compiler invocations (does @pitrou or @kou know how they were determined in commits like these?). Ideally the linker would do the dead section elimination, but I didn't dig into why the build system wouldn't allow so (I'm guessing the libs have to be bundled in and linked separately somehow?).

I'm hoping to upgrade the dependency when there is code using it (which would be the GCS implementation of the Arrow file interfaces).

I determined required abseil libraries manually. CI error helped me.

salqadri · 2021-07-30T22:02:45Z

What's the status of this PR?

josiahyan · 2021-08-01T03:10:10Z

What's the status of this PR?

Sorry, but I lost track of following up on this PR! I believed it was ready to merge in Feb, but assumed people had lost interest in pursuing (native) GCS support, or that this change was not quite worth making for now. Is there still a need for this? If so, I can attempt to rebase it again.

nealrichardson · 2021-08-01T12:29:04Z

What's the status of this PR?

Sorry, but I lost track of following up on this PR! I believed it was ready to merge in Feb, but assumed people had lost interest in pursuing (native) GCS support, or that this change was not quite worth making for now. Is there still a need for this? If so, I can attempt to rebase it again.

I think there's interest (I'm interested, at least). I'd like to see someone more skilled in cmake than I to review and approve: @kou @pitrou @xhochy ?

kou · 2021-08-01T20:46:14Z

Could you rebase on the master? Then I'll review this.

josiahyan · 2021-08-01T20:47:15Z

Will rebase!

…

On Sun, Aug 1, 2021 at 1:46 PM Sutou Kouhei ***@***.***> wrote: Could you rebase on the master? Then I'll review this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8757 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPH2DGIQYWZJBQFOK4RMNLT2WXCDANCNFSM4UA73NKQ> .

emkornfield · 2021-08-20T23:45:40Z

@josiahyan do you think you will have time to rebase? Otherwise I can see how difficult this might be.

…pt 2020, Patch 2

josiahyan · 2021-08-23T05:45:44Z

@josiahyan do you think you will have time to rebase? Otherwise I can see how difficult this might be.

Whoops sorry, I lost track of this. I’ll be trying to rebase this - I don’t see any major changes (so far) that should result in conflicts.

emkornfield · 2021-09-12T22:40:32Z

 autoconf
 ccache
+google-cloud-cpp=1.20.0
+nlohmann_json>=3.4.0


what is the new JSON dependency for?

I guess google's SDK must rely on it?

I am one of the authors of google's GCS SDK. Yes, the SDK definitely depends on nlohmann_json.

Conda has dependency handling, so there is no reason to mention it explicitly, though.

coryan · 2021-09-27T17:07:39Z

+       GCPSDK_PREFIX_PATH)
+
+  set(GCPSDK_CXX_FLAGS "${EP_CXX_FLAGS}")
+  # workaround for crc32c not declaring its header interface in its Find*.cmake file


You can avoid this by using crc32c 1.1.1:

google/crc32c@b377ce4

josiahyan · 2021-09-28T16:56:45Z

I’ve discussed this PR with @coryan , and we think its better to start over, given the need to upgrade dependencies, and handle the Abseil libraries (which I believe, when used with gRPC are causing the build to fail on some platforms) with more scripting. He will be taking over this work.

github-actions Bot added the Component: C++ label Nov 24, 2020

josiahyan marked this pull request as draft November 24, 2020 15:27

josiahyan marked this pull request as ready for review November 25, 2020 04:31

pitrou reviewed Dec 1, 2020

View reviewed changes

Comment thread ci/docker/conda-cpp.dockerfile Outdated

josiahyan force-pushed the ARROW-8147-add-gcp-sdk branch from 7d1014a to 4fd6a36 Compare December 1, 2020 14:43

josiahyan force-pushed the ARROW-8147-add-gcp-sdk branch 2 times, most recently from 979c43a to 953de85 Compare December 1, 2020 16:43

nealrichardson reviewed Dec 1, 2020

View reviewed changes

Comment thread cpp/thirdparty/versions.txt Outdated

josiahyan force-pushed the ARROW-8147-add-gcp-sdk branch from 3c26730 to b9313da Compare December 1, 2020 21:21

josiahyan requested review from nealrichardson and pitrou December 4, 2020 14:37

github-actions Bot added the needs-rebase A PR that needs to be rebased by the author label Dec 24, 2020

jorgecarleitao force-pushed the master branch from d4608a9 to 356c300 Compare February 14, 2021 12:09

josiahyan force-pushed the ARROW-8147-add-gcp-sdk branch from b9313da to 850c239 Compare February 28, 2021 01:03

josiahyan marked this pull request as draft February 28, 2021 01:03

josiahyan marked this pull request as ready for review February 28, 2021 04:29

xhochy reviewed Feb 28, 2021

View reviewed changes

josiahyan added 9 commits August 23, 2021 04:26

ARROW-8147: [C++] Add google-cloud-cpp to ThirdpartyToolchain

6600fb5

Add google-cloud-cpp to conda requirements for CI

17d3636

Add dependency on nlohmann_json for google-cloud-cpp build

72cb6f0

Move google-cloud-cpp conda dependencies to unix only

69b5799

Reorder dependencies, minor fixes

433eca0

Add URLs to fetch dependencies to cpp/thirdparty/versions.txt

46e9dcc

Lock GCP SDK version to 1.20.0

99852f9

Remove GCS SDK from Windows cpp appveyor CI build

cb06b47

Resynchronize GCS SDK's Abseil includes against Abseil LTS branch, Se…

7531ca5

…pt 2020, Patch 2

josiahyan force-pushed the ARROW-8147-add-gcp-sdk branch from 7076b4c to 7531ca5 Compare August 23, 2021 05:44

Fix typo

d92433a

josiahyan marked this pull request as draft August 23, 2021 07:25

emkornfield reviewed Sep 12, 2021

View reviewed changes

coryan reviewed Sep 27, 2021

View reviewed changes

josiahyan closed this Sep 28, 2021

asfimport mentioned this pull request Oct 5, 2021

[C++] Add google-cloud-cpp to ThirdpartyToolchain #24352

Closed

Uh oh!

Conversation

josiahyan commented Nov 24, 2020

Uh oh!

josiahyan commented Nov 24, 2020

Uh oh!

github-actions Bot commented Nov 24, 2020

Uh oh!

xhochy commented Nov 24, 2020

Uh oh!

josiahyan commented Nov 25, 2020

Uh oh!

josiahyan commented Nov 25, 2020

Uh oh!

josiahyan commented Nov 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xhochy commented Nov 25, 2020

Uh oh!

josiahyan commented Nov 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

josiahyan commented Dec 1, 2020

Uh oh!

Uh oh!

nealrichardson commented Jan 29, 2021

Uh oh!

josiahyan commented Jan 29, 2021 via email

Uh oh!

josiahyan commented Feb 28, 2021

Uh oh!

xhochy Feb 28, 2021

Choose a reason for hiding this comment

Uh oh!

josiahyan Feb 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou Feb 28, 2021

Choose a reason for hiding this comment

Uh oh!

salqadri commented Jul 30, 2021

Uh oh!

josiahyan commented Aug 1, 2021

Uh oh!

nealrichardson commented Aug 1, 2021

Uh oh!

kou commented Aug 1, 2021

Uh oh!

josiahyan commented Aug 1, 2021 via email

Uh oh!

emkornfield commented Aug 20, 2021

Uh oh!

josiahyan commented Aug 23, 2021

Uh oh!

emkornfield Sep 12, 2021

Choose a reason for hiding this comment

Uh oh!

emkornfield Sep 12, 2021

Choose a reason for hiding this comment

Uh oh!

coryan Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

pitrou Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

coryan Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

josiahyan commented Sep 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

josiahyan commented Nov 25, 2020 •

edited

Loading

josiahyan commented Nov 25, 2020 •

edited

Loading

josiahyan Feb 28, 2021 •

edited

Loading