HDDS-15317. Hadoop 3.4.3 libhaoop library for Mac-OS aarch64 platform

[ChenSammi]

What changes were proposed in this pull request?

add Hadoop 3.4.3 libhaoop library for Mac-OS aarch64 platform

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15317

How was this patch tested?

libhadoop.1.0.0.dylib and libhadoop.dylib load and APIs are test locally with a UT test in ozone.

[ChenSammi]

@adoroszlai , could you help with the review?

[adoroszlai]

@ChenSammi The docker image runs Linux, not macOS. It already has libhadoop for Linux (both x64 and aarch64). Why is Mac library needed and how would it be used?

We should upgrade to Hadoop 3.4.3 instead:

ozone-docker-runner/Dockerfile

Line 130 in 882ae82

hadoop_version=3.4.2 ; \

We also need to remove -lean from tarball name:

ozone-docker-runner/Dockerfile

Lines 132 to 133 in 882ae82

	x86_64) file=hadoop-${hadoop_version}-lean.tar.gz ;; \
	aarch64) file=hadoop-${hadoop_version}-aarch64-lean.tar.gz ;; \

because Hadoop release no longer comes with full/lean variants since 3.4.3.

[ChenSammi]

@adoroszlai , it's for apache/ozone#10305.

[adoroszlai]

it's for apache/ozone#10305.

I see, but that doesn't explain why it should be part of the Docker image.

[ChenSammi]

it's for apache/ozone#10305.

I see, but that doesn't explain why it should be part of the Docker image.

I need a public place to host the hadoop native library for Mac-OS, so that apache/ozone#10305 can download from. Do you have other suggestion better than host in this repo? I'm open to it.

[adoroszlai]

I need a public place to host the hadoop native library for Mac-OS, so that apache/ozone#10305 can download from. Do you have other suggestion better than host in this repo? I'm open to it.

Thank you for the openness.

Ideally Hadoop should release these native libs in platform-specific jars, e.g. Netty does that.

Until then I suggest creating a separate repository (maybe ozone-thirdparty) for these artifacts. I would store them in the dir native/hadoop-3.4.3 in case we may want to add Java libraries in the future. We should also add the same libs for Linux so that we can avoid download of Hadoop tarballs. Something like:

libhadoop-linux-x86_64.so.1.0.0
libhadoop-linux-aarch_64.so.1.0.0
libhadoop-osx-aarch_64.1.0.0.dylib

BTW, do I understand correctly that this Mac library is built locally, not part of the official Hadoop release? If so, we should mention it in a README.txt in the same dir, maybe with instructions on how to build it if needed (e.g. for new Hadoop versions).

[ChenSammi]

I need a public place to host the hadoop native library for Mac-OS, so that apache/ozone#10305 can download from. Do you have other suggestion better than host in this repo? I'm open to it.

Thank you for the openness.

Ideally Hadoop should release these native libs in platform-specific jars, e.g. Netty does that.

Until then I suggest creating a separate repository (maybe ozone-thirdparty) for these artifacts. I would store them in the dir native/hadoop-3.4.3 in case we may want to add Java libraries in the future. We should also add the same libs for Linux so that we can avoid download of Hadoop tarballs. Something like:

libhadoop-linux-x86_64.so.1.0.0
libhadoop-linux-aarch_64.so.1.0.0
libhadoop-osx-aarch_64.1.0.0.dylib

BTW, do I understand correctly that this Mac library is built locally, not part of the official Hadoop release? If so, we should mention it in a README.txt in the same dir, maybe with instructions on how to build it if needed (e.g. for new Hadoop versions).

Yes, Hadoop release doesn't provide native library for Mac platform. This is built on my local environment.
The hadoop native build on Mac is a little bit complex. Due to lack of some Linux headers referred by YARN container native code, it cannot succeed completely, but enough to get the libhadoop native library.
And I hit several issues during that. And these issues varies on different Mac OS kernel version, cmake, gcc, zlib, openssl, and hadoop verions ( I have two Macbook, one intel chipset, one arm chipset). So I don't have enough confidence currently to provide a official hadoop native build guide for Mac platform.

[ChenSammi]

ozone-thirdparty is kind of a good idea. Do you know how to create an apache repo, shall I raise a ticket to apache infra team?

[adoroszlai]

Do you know how to create an apache repo,

"Projects can request new, blank, public repositories through selfserve" (source)

[ChenSammi]

https://github.com/apache/ozone-thirdparty

[ChenSammi]

@adoroszlai , can you help review this one apache/ozone-thirdparty#1?