[feat][build] Support ARM64-based docker images by tisonkun · Pull Request #17733 · apache/pulsar

tisonkun · 2022-09-20T07:30:53Z

This refers to #12944.

Verifying this change

mvn clean install -DskipTests
./build/build_java_test_image.sh                                                
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DpythonClientBuildArch=aarch64

Documentation

doc-required
(Your PR needs to update docs and you will update later)
doc-not-needed
(Please explain why)
doc
(Your PR contains doc changes)
doc-complete
(Docs have been already added)

Matching PR in forked repository

PR in forked repository: tisonkun#2

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2022-09-20T08:42:40Z

cc @nodece @BewareMyPower @merlimat @lhotari @michaeljmarshall @heesung-sn

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2022-09-20T11:14:39Z

-     && apt-get -y --purge autoremove \
+
+# Cleanup apt
+RUN apt-get -y --purge autoremove \
     && apt-get autoclean \
     && apt-get clean \
-     && rm -rf /var/lib/apt/lists/* \
-     && pip3 install pyyaml==5.4.1
+     && rm -rf /var/lib/apt/lists/*
+
+RUN pip3 install pyyaml==5.4.1


I don't know why. But this change causes local build fails with returned a non-zero code: 100. So keep it as is and revert intermediate changes.

merlimat · 2022-09-20T15:13:59Z

+
+# TODO: remove these lines once grpcio doesn't need to compile from source on ARM64 platform
+apt update
+apt -y install build-essential python3-dev


This would be increasing the size of the pulsar image.

I could see 2 different options to avoid it:

Only install compilers on ARM64 build (at least we don't make the x86_64 image any worse)

Do the Grpc pip install in a separate Docker stage and then only copy the built wheel file. (this should work for both arm & x86)

Resolved by option 1 at 2793885.

Option 2 seems also to require installing build-essential python3-dev for pip install grpcio on ARM64 platform? Conditional installation is easier to perform inside a script than inside the Dockerfile AFAIK...

Signed-off-by: tison <wander4096@gmail.com>

heesung-sohn · 2022-09-26T22:42:25Z

+
 PYTHON_MAJOR_MINOR=$(python3 -V | sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/')
 WHEEL_FILE=$(ls /pulsar/pulsar-client | grep "cp${PYTHON_MAJOR_MINOR}")
 pip3 install /pulsar/pulsar-client/${WHEEL_FILE}[all]


[INFO] DOCKER> [91m++ python3 -V ++ sed -E 's/.* ([[:digit:]]+)\.([[:digit:]]+).*/\1\2/' [INFO] DOCKER> [91m+ PYTHON_MAJOR_MINOR=38 [INFO] DOCKER> [91m++ ls /pulsar/pulsar-client ++ grep cp38 [INFO] DOCKER> [91m+ WHEEL_FILE='pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl' [INFO] DOCKER> [91m+ pip3 install /pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl 'pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]' [INFO] DOCKER> [91mERROR: pulsar_client-2.11.0-cp38-cp38-manylinux1_x86_64.whl is not a supported wheel on this platform. [INFO] DOCKER> Removing intermediate container a88eb585f4d1 [ERROR] DOCKER> Unable to build image [apachepulsar/pulsar] : "The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ["The command '/bin/sh -c /pulsar/bin/install-pulsar-client.sh' returned a non-zero code: 1" ]

observed this error.

this problem will anyway go away once we make the changes for PIP-209 (moving C++ client out of repo)

@heesung-sn

mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DpythonClientBuildArch=aarch64

Be aware that you should pass -DpythonClientBuildArch=aarch64.

I made this patch to support ARM64-based docker images while keeping the current workflow as is. That is, you should manually specify that you want to build an ARM64-based image.

About automatically detecting platforms, see also #17733 (comment) that we have some name matching problems.

And a clean solution would be, as @merlimat described, we publish C++/Python client alone and download them from the package manager so that leveraging the package manager's platform detect functionality.

I provided the pythonClientBuildArch option when I encountered the error.

mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests -DpythonClientBuildArch=aarch64

I tried to hard-code like the below, then it passes. I wonder if the client somehow output wheel files for both archs.

pip3 install /pulsar/pulsar-client/pulsar_client-2.11.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl[all]

(cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com>

* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3) * PIP-155: Removed Python 2 support (apache#15376) * Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <mmarshall@apache.org> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][docker] Switch to Temurin JDK (apache#17129) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 4378856) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Support ARM64-based docker images (apache#17733) (cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * PIP-209: Removed C++/Python clients from main repo (apache#17881) * PIP-209: Removed C++/Python clients from main repo * Removed python directory from Docekrfile * Fixed python client version argument scoping * Fixed handling of pulsar.functions.serde (cherry picked from commit f3c547b) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][build] Avoid building image multiple times (apache#17208) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 79a97a9) * [improve] Allow to build and push multi-arch Docker images (apache#19432) Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zixuan Liu <nodeces@gmail.com> Co-authored-by: tison <wander4096@gmail.com> (cherry picked from commit 4190e40) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix publish image script (apache#20305) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 94c7bf3) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435) Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit d7f3558) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Adapt to Python client to be compatible with ARM arch Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550) (cherry picked from commit 05f7e62) * [improve][misc] Include native epoll library for Netty for arm64 From apache#22319 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][misc] Rename all shaded Netty native libraries From apache#22415 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [cleanup][build] Cleanup -Ddocker.nocache=true Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix ubuntu mirror Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix license Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Downgrade docker-maven to 0.43.3 Signed-off-by: Zixuan Liu <nodeces@gmail.com> --------- Signed-off-by: Zixuan Liu <nodeces@gmail.com> Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Matteo Merli <mmerli@apache.org> Co-authored-by: Michael Marshall <mmarshall@apache.org> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org>

tisonkun added 3 commits September 20, 2022 14:08

detect arch for Golang installation

6a2d129

Signed-off-by: tison <wander4096@gmail.com>

detect arch for JDK installation

e0d465a

Signed-off-by: tison <wander4096@gmail.com>

fix python client build on ARM64

b2d5605

Signed-off-by: tison <wander4096@gmail.com>

tisonkun force-pushed the build-on-arm64 branch from 5b4f148 to b2d5605 Compare September 20, 2022 07:52

fix java_home path

9fe71d9

Signed-off-by: tison <wander4096@gmail.com>

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Sep 20, 2022

fix resolve architecture

a13e8cb

Signed-off-by: tison <wander4096@gmail.com>

BewareMyPower assigned tisonkun Sep 20, 2022

BewareMyPower added area/build cl. labels Sep 20, 2022

tisonkun mentioned this pull request Sep 20, 2022

[improve][docker] Switch to Temurin JDK #17129

Merged

4 tasks

BewareMyPower removed the cl. label Sep 20, 2022

tisonkun mentioned this pull request Sep 20, 2022

ARM based docker image #12944

Closed

revert chained run commands

1ae7709

Signed-off-by: tison <wander4096@gmail.com>

tisonkun changed the title ~~Docker images build on arm64~~ [feat][build] Support ARM64-based docker images Sep 20, 2022

tisonkun commented Sep 20, 2022

View reviewed changes

Comment thread docker/pulsar/pom.xml

merlimat reviewed Sep 20, 2022

View reviewed changes

tisonkun added 2 commits September 20, 2022 23:42

architecture -> ARCH

1909f25

Signed-off-by: tison <wander4096@gmail.com>

install build-essential python3-dev only for ARM64

2793885

Signed-off-by: tison <wander4096@gmail.com>

merlimat approved these changes Sep 20, 2022

View reviewed changes

merlimat added this to the 2.12.0 milestone Sep 20, 2022

BewareMyPower approved these changes Sep 20, 2022

View reviewed changes

Technoboy- merged commit 9a2aeb2 into apache:master Sep 21, 2022

michaeljmarshall approved these changes Sep 21, 2022

View reviewed changes

tisonkun deleted the build-on-arm64 branch September 21, 2022 03:28

heesung-sohn reviewed Sep 26, 2022

View reviewed changes

lhotari mentioned this pull request Apr 2, 2024

[Bug] NumberFormatException when consuming messages from pulsar-admin CLI #22405

Closed

3 tasks

nodece pushed a commit to nodece/pulsar that referenced this pull request Apr 28, 2024

[feat][build] Support ARM64-based docker images (apache#17733)

441001b

(cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feat][build] Support ARM64-based docker images - #17733

[feat][build] Support ARM64-based docker images#17733
Technoboy- merged 8 commits into
apache:masterfrom
tisonkun:build-on-arm64

tisonkun commented Sep 20, 2022 •

edited

Loading

Uh oh!

tisonkun commented Sep 20, 2022 •

edited

Loading

Uh oh!

tisonkun Sep 20, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merlimat Sep 20, 2022

Uh oh!

tisonkun Sep 20, 2022

Uh oh!

Uh oh!

Uh oh!

heesung-sohn Sep 26, 2022

Uh oh!

merlimat Sep 26, 2022

Uh oh!

tisonkun Sep 26, 2022

Uh oh!

tisonkun Sep 26, 2022 •

edited

Loading

Uh oh!

heesung-sohn Sep 26, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

tisonkun commented Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verifying this change

Documentation

Matching PR in forked repository

Uh oh!

tisonkun commented Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tisonkun Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merlimat Sep 20, 2022

Choose a reason for hiding this comment

Uh oh!

tisonkun Sep 20, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

heesung-sohn Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

merlimat Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

tisonkun Sep 26, 2022

Choose a reason for hiding this comment

Uh oh!

tisonkun Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

heesung-sohn Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tisonkun commented Sep 20, 2022 •

edited

Loading

tisonkun commented Sep 20, 2022 •

edited

Loading

tisonkun Sep 20, 2022 •

edited

Loading

tisonkun Sep 26, 2022 •

edited

Loading

heesung-sohn Sep 26, 2022 •

edited

Loading