[fix][cpp] Fix libcurl build failure when building deb package - #17614
Merged
lhotari merged 5 commits intoSep 13, 2022
Conversation
### Motivation See apache#17538 (comment) The root cause is when libcurl is built from source, it uses [`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so` links to the correct dependencies in runtime. In Linux, a dynamic library links to the paths of `/etc/ld.so.conf` by default. However, different from other images like `centos:7` and `alpine`, this file includes `/usr/lib/x86_64-linux-gnu` in `debian:9`. ```bash $ cat /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf $ cat /etc/ld.so.conf.d/*.conf /usr/lib/x86_64-linux-gnu/libfakeroot # libc default configuration /usr/local/lib # Multiarch support /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ``` When libcurl is compiled, it links to the install path of libopenssl via the `--with-ssl` option: https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85 i.e. `/usr/local/ssl/lib/libopenssl.so`. However, after the `libcurl.so` is built, it links to `/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output: ```bash $ ldd /usr/local/lib/libcurl.so /usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so) ``` In `debian:9`, the default libopenssl version is 1.1.0: ```bash $ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL OpenSSL 1.1.0l 10 Sep 2019 ``` The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see https://abi-laboratory.pro/index.php?view=timeline&l=openssl. ### Modifications Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to build deb package. Actually it's not required for other images like `centos:7`, but it's also good to add the `LD_LIBRARY_PATH` to them. So this PR set the environment variable to them as well.
BewareMyPower
added
type/bug
BewareMyPower
requested review from
RobertIndie,
codelipenghui,
lhotari and
merlimat
September 13, 2022 10:29
lhotari
approved these changes
Sep 13, 2022
lhotari
left a comment
lhotari
left a comment
Member
There was a problem hiding this comment.
Thank you. Great explanations too.
nicoloboschi
approved these changes
Sep 13, 2022
4 tasks
RobertIndie
approved these changes
Sep 13, 2022
Member
|
This PR fixes the broken master branch. The GitHub Actions workflow was also broken in a recently merged change (#17571 |
tisonkun
pushed a commit
to tisonkun/pulsar
that referenced
this pull request
Sep 14, 2022
…e#17614) * [fix][cpp] Fix libcurl build failure when building deb package ### Motivation See apache#17538 (comment) The root cause is when libcurl is built from source, it uses [`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so` links to the correct dependencies in runtime. In Linux, a dynamic library links to the paths of `/etc/ld.so.conf` by default. However, different from other images like `centos:7` and `alpine`, this file includes `/usr/lib/x86_64-linux-gnu` in `debian:9`. ```bash $ cat /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf $ cat /etc/ld.so.conf.d/*.conf /usr/lib/x86_64-linux-gnu/libfakeroot # libc default configuration /usr/local/lib # Multiarch support /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ``` When libcurl is compiled, it links to the install path of libopenssl via the `--with-ssl` option: https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85 i.e. `/usr/local/ssl/lib/libopenssl.so`. However, after the `libcurl.so` is built, it links to `/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output: ```bash $ ldd /usr/local/lib/libcurl.so /usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so) ``` In `debian:9`, the default libopenssl version is 1.1.0: ```bash $ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL OpenSSL 1.1.0l 10 Sep 2019 ``` The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see https://abi-laboratory.pro/index.php?view=timeline&l=openssl. ### Modifications Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to build deb package. Actually it's not required for other images like `centos:7`, but it's also good to add the `LD_LIBRARY_PATH` to them. So this PR set the environment variable to them as well. * Fix workflow so that cpp-tests isn't skipped * Revisit workflow fix to cover doc only workflows too * Fix multi-line condition Co-authored-by: Lari Hotari <lhotari@apache.org>
4 tasks
Jason918
pushed a commit
that referenced
this pull request
Sep 17, 2022
* [fix][cpp] Fix libcurl build failure when building deb package See #17538 (comment) The root cause is when libcurl is built from source, it uses [`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so` links to the correct dependencies in runtime. In Linux, a dynamic library links to the paths of `/etc/ld.so.conf` by default. However, different from other images like `centos:7` and `alpine`, this file includes `/usr/lib/x86_64-linux-gnu` in `debian:9`. ```bash $ cat /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf $ cat /etc/ld.so.conf.d/*.conf /usr/lib/x86_64-linux-gnu/libfakeroot /usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ``` When libcurl is compiled, it links to the install path of libopenssl via the `--with-ssl` option: https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85 i.e. `/usr/local/ssl/lib/libopenssl.so`. However, after the `libcurl.so` is built, it links to `/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output: ```bash $ ldd /usr/local/lib/libcurl.so /usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so) ``` In `debian:9`, the default libopenssl version is 1.1.0: ```bash $ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL OpenSSL 1.1.0l 10 Sep 2019 ``` The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see https://abi-laboratory.pro/index.php?view=timeline&l=openssl. Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to build deb package. Actually it's not required for other images like `centos:7`, but it's also good to add the `LD_LIBRARY_PATH` to them. So this PR set the environment variable to them as well. * Fix workflow so that cpp-tests isn't skipped * Revisit workflow fix to cover doc only workflows too * Fix multi-line condition Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit 0754ea1)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Sep 20, 2022
…e#17614) * [fix][cpp] Fix libcurl build failure when building deb package See apache#17538 (comment) The root cause is when libcurl is built from source, it uses [`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so` links to the correct dependencies in runtime. In Linux, a dynamic library links to the paths of `/etc/ld.so.conf` by default. However, different from other images like `centos:7` and `alpine`, this file includes `/usr/lib/x86_64-linux-gnu` in `debian:9`. ```bash $ cat /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf $ cat /etc/ld.so.conf.d/*.conf /usr/lib/x86_64-linux-gnu/libfakeroot /usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ``` When libcurl is compiled, it links to the install path of libopenssl via the `--with-ssl` option: https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85 i.e. `/usr/local/ssl/lib/libopenssl.so`. However, after the `libcurl.so` is built, it links to `/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output: ```bash $ ldd /usr/local/lib/libcurl.so /usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so) ``` In `debian:9`, the default libopenssl version is 1.1.0: ```bash $ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL OpenSSL 1.1.0l 10 Sep 2019 ``` The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see https://abi-laboratory.pro/index.php?view=timeline&l=openssl. Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to build deb package. Actually it's not required for other images like `centos:7`, but it's also good to add the `LD_LIBRARY_PATH` to them. So this PR set the environment variable to them as well. * Fix workflow so that cpp-tests isn't skipped * Revisit workflow fix to cover doc only workflows too * Fix multi-line condition Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit 0754ea1) (cherry picked from commit 7162c30)
tisonkun
added a commit
to tisonkun/pulsar
that referenced
this pull request
Oct 4, 2022
This OR logic has been introduced by apache#17614. I don't know why it's added in the first place - we shouldn't have a OR here. But anyway since pulsar-client-cpp moved out, it should be reasonable to fix the workflow.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
See #17538 (comment)
The root cause is when libcurl is built from source, it uses
ldto check if thelibcurl.solinks to the correct dependencies in runtime. In Linux, a dynamic library links to the paths of/etc/ld.so.confby default. However, different from other images likecentos:7andalpine, this file includes/usr/lib/x86_64-linux-gnuindebian:9.When libcurl is compiled, it links to the install path of libopenssl via the
--with-ssloption:pulsar/pulsar-client-cpp/pkg/deb/Dockerfile
Line 85 in 1f50366
i.e.
/usr/local/ssl/lib/libopenssl.so.However, after the
libcurl.sois built, it links to/usr/lib/x86_64-linux-gnu/libssl.so.1.1, see the following output:$ ldd /usr/local/lib/libcurl.so /usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so)In
debian:9, the default libopenssl version is 1.1.0:$ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL OpenSSL 1.1.0l 10 Sep 2019The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see https://abi-laboratory.pro/index.php?view=timeline&l=openssl.
Modifications
Set the
LD_LIBRARY_PATHto/usr/local/ssl/libin the Dockerfile to build deb package.In Linux, a dynamic library links to the
LD_LIBRARY_PATHin prior to/etc/ld.so.conf.We can see the order in the man page of
ld.Actually it's not required for other images like
centos:7, but it's also good to add theLD_LIBRARY_PATHto them. So this PR set the environment variable to them as well.Documentation
doc-required(Your PR needs to update docs and you will update later)
doc-not-needed(Please explain why)
doc(Your PR contains doc changes)
doc-complete(Docs have been already added)