HDDS-15318. Download libhadoop library during maven build by ChenSammi · Pull Request #10305 · apache/ozone

ChenSammi · 2026-05-19T05:05:43Z

What changes were proposed in this pull request?

download libhadoop library during maven build, for local unit test, and dist package.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15318

How was this patch tested?

Test locally.

mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pnative-mac -Pdist

sammi@X3HF9K66K7 target % find . -name libhadoop* 
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.dylib
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib

sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x  0 sammi  staff        0 May 18 20:30 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop.1.0.0.dylib
-rw-r--r--  0 sammi  staff   157368 May 18 20:30 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.1.0.0.dylib

mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pnative -Pdist

sammi@X3HF9K66K7 target % find . -name libhadoop*
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so
./ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x  0 sammi  staff        0 May 19 12:02 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop.so.1.0.0
-rwxr-xr-x  0 sammi  staff   890256 May 19 12:02 ozone-2.2.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0


mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist

sammi@X3HF9K66K7 target % find . -name libhadoop*                             
sammi@X3HF9K66K7 target % tar -tvf ozone-2.2.0-SNAPSHOT.tar.gz| grep libhadoop

The downloaded file in cached in maven local repo, so only download once if it's not changed.

sammi@X3HF9K66K7 target % ls -al ~/.m2/repository/.cache/download-maven-plugin
total 1006024
drwxr-xr-x  5 sammi  staff        160 May 18 19:47 .
drwxr-xr-x  3 sammi  staff         96 May 18 10:56 ..
-rw-r--r--  1 sammi  staff  514917037 May 18 11:50 hadoop-3.4.3.tar.gz_0deb1bfef085d775400e262f41867f8e
-rw-r--r--  1 sammi  staff        478 May 18 19:47 index.ser
-rw-r--r--  1 sammi  staff     157368 May 18 19:47 libhadoop.1.0.0.dylib_1aa24cad2f558d1e1012dfd83037efc1

jojochuang

thanks for adding this!

jojochuang · 2026-05-19T06:26:51Z

+  run mkdir -p ./lib/native
+
+  # Copy the files and symlinks safely
+  run cp -rP "${NATIVE_LIBS_DIR}/"libhadoop.* ./lib/native


will the ozone-env.sh script (or docker-config for our Docker deployments) need to set NATIVE_LIBS_DIR to lib/native for ozone to pick up libhadoop from this path?

./lib/native doesn't always exists. But yes, I think ozone-env.sh or ozone.sh can auto add lib/native to java library path if it exists.

Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

adoroszlai

Thanks @ChenSammi for the patch.

adoroszlai · 2026-05-19T08:08:34Z

+                </goals>
+                <phase>generate-resources</phase>
+                <configuration>
+                  <url>https://archive.apache.org/dist/hadoop/common/hadoop-${hadoop.version}/hadoop-${hadoop.version}.tar.gz</url>


We should not download 500MB tarball for each build. Need to find a way to get these libs with less overhead.

It only download once for every hadoop version locally. Maven will cache it. See above "How was this patch tested" section. And if -Pnative or -Pnative-mac is not enabled on maven build, it will not trigger the download.

adoroszlai · 2026-05-19T08:13:39Z

+          <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-antrun-plugin</artifactId>
+            <version>3.1.0</version>


Suggested change

<version>3.1.0</version>

adoroszlai · 2026-05-19T08:14:03Z

+          <plugin>
+            <groupId>com.googlecode.maven-download-plugin</groupId>
+            <artifactId>download-maven-plugin</artifactId>
+            <version>1.6.8</version>


Version (newer one) already defined.

Suggested change

<version>1.6.8</version>

It's this version on feature branch. Next time, when merge master, it will be updated with master branch version.

adoroszlai · 2026-05-19T08:14:15Z

+          <plugin>
+            <groupId>com.googlecode.maven-download-plugin</groupId>
+            <artifactId>download-maven-plugin</artifactId>
+            <version>1.6.8</version>


Suggested change

<version>1.6.8</version>

adoroszlai · 2026-05-19T08:15:44Z

+          <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-antrun-plugin</artifactId>
+            <version>3.1.0</version>


Suggested change

<version>3.1.0</version>

adoroszlai · 2026-05-19T08:32:14Z

+      <id>native</id>
+      <activation>
+        <os>
+          <family>linux</family>
+          <arch>x86_64</arch>
+        </os>
+        <property>
+          <name>fetch-native-hadoop</name>
+        </property>
+      </activation>


Release manager may be using Mac or Linux, but needs to create package with lib for all supported platforms. I think these download/extract steps should be moved to existing dist profile (preferably in hadoop-ozone/dist/pom.xml).

I did put it into dist pom.xml in first version. But there are at least two more packages (unit test and integration test) requires native library to run(not covered in this patch). So to avoid duplicate the profiles in multiple pom.xml, gemini suggests to put it in root pom.xml, and with inherited false.

OK, if it's needed for tests, then keep it in root pom.xml, still, it should be platform-agnostic.

What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?

What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?

Build "needs to create package with lib for all supported platforms" regardless of which OS/arch it runs on.

If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean? Or what you mean is we have two dist ozone-2.1.0-SNAPSHOT directory, for example ozone-2.1.0-SNAPSHOT-x86-64, and ozone-2.1.0-SNAPSHOT-aarch64?

I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.

If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean?

Yes.

I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.

OK. (We do include some Mac/Windows native libs for Netty.)

mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist,native,native-mac sammi@X3HF9K66K7 target % ls -al ./ozone-2.1.0-SNAPSHOT/lib/native/ total 2056 drwxr-xr-x 6 sammi staff 192 May 19 23:17 . drwxr-xr-x 3 sammi staff 96 May 19 23:17 .. lrwxr-xr-x 1 sammi staff 28 May 19 23:17 libhadoop.dylib -> libhadoop_osx_aarch_64.dylib lrwxr-xr-x 1 sammi staff 25 May 19 23:17 libhadoop.so -> libhadoop_linux_x86_64.so -rw-r--r-- 1 sammi staff 890256 May 19 23:17 libhadoop_linux_x86_64.so -rw-r--r-- 1 sammi staff 157368 May 19 23:17 libhadoop_osx_aarch_64.dylib sammi@X3HF9K66K7 target % tar -tvf ozone-2.1.0-SNAPSHOT.tar.gz| grep libhadoop lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop_osx_aarch_64.dylib -rw-r--r-- 0 sammi staff 157368 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_osx_aarch_64.dylib -rw-r--r-- 0 sammi staff 890256 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_linux_x86_64.so lrwxr-xr-x 0 sammi staff 0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop_linux_x86_64.so

…rsion

HDDS-15318. Download libhadoop library during maven build

1eae53a

ChenSammi requested review from adoroszlai and jojochuang May 19, 2026 05:06

jojochuang reviewed May 19, 2026

View reviewed changes

jojochuang requested a review from Copilot May 19, 2026 06:28

Copilot started reviewing on behalf of jojochuang May 19, 2026 06:28 View session

Update hadoop-ozone/dist/dev-support/bin/dist-layout-stitching

b69984c

Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com>

Copilot AI reviewed May 19, 2026

View reviewed changes

adoroszlai marked this pull request as draft May 19, 2026 08:14

adoroszlai requested changes May 19, 2026

View reviewed changes

ChenSammi mentioned this pull request May 19, 2026

HDDS-15317. Hadoop 3.4.3 libhaoop library for Mac-OS aarch64 platform apache/ozone-docker-runner#66

Closed

use mvn property for download-maven-plugin and maven-antrun-plugin ve…

62af7c0

…rsion

ChenSammi requested a review from adoroszlai May 19, 2026 09:15

ChenSammi marked this pull request as ready for review May 19, 2026 15:27

download library from ozone-thirdparty

8aca9a4

ChenSammi force-pushed the HDDS-15318 branch from 453e74a to 8aca9a4 Compare May 19, 2026 15:29

Conversation

ChenSammi commented May 19, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai May 19, 2026 • edited by ChenSammi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChenSammi May 19, 2026 •

edited

Loading

adoroszlai May 19, 2026 •

edited by ChenSammi

Loading

adoroszlai May 19, 2026 •

edited

Loading

ChenSammi May 19, 2026 •

edited

Loading

ChenSammi May 19, 2026 •

edited

Loading

ChenSammi May 19, 2026 •

edited

Loading