Skip to content

HDDS-15318. Download libhadoop library during maven build #10305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChenSammi merged 4 commits into from
May 20, 2026
Merged

HDDS-15318. Download libhadoop library during maven build #10305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1eae53a
HDDS-15318. Download libhadoop library during maven build
ChenSammi May 18, 2026
b69984c
Update hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
ChenSammi May 19, 2026
62af7c0
use mvn property for download-maven-plugin and maven-antrun-plugin ve…
ChenSammi May 19, 2026
8aca9a4
download library from ozone-thirdparty
ChenSammi May 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,24 @@ for file in $(find "${ROOT}" -path '*/share/ozone/lib/*jar' | sort); do
cp -p "$file" share/ozone/lib/
done

# ---------------------------------------------------------
# Copy Hadoop Native Libraries (libhadoop.so) - Conditionally
# ---------------------------------------------------------
NATIVE_LIBS_DIR="${ROOT}/target/native-lib"

# Check if the libhadoop files actually exist before attempting to copy
if ls "${NATIVE_LIBS_DIR}"/libhadoop* 1> /dev/null 2>&1; then
echo "Found Hadoop native libraries. Copying to distribution..."

# Create the native directory in the final staging area
run mkdir -p ./lib/native

# Copy the files and symlinks safely
run cp -rP "${NATIVE_LIBS_DIR}/"libhadoop* ./lib/native
else
echo "Hadoop native libraries not found. Skipping native copy."
fi

#workaround for https://issues.apache.org/jira/browse/MRESOURCES-236
find ./compose -name "*.sh" -exec chmod 755 {} \;
find ./kubernetes -name "*.sh" -exec chmod 755 {} \;
120 changes: 120 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2633,5 +2633,125 @@
</plugins>
</build>
</profile>
<profile>
<id>native</id>
<activation>
<os>
<family>linux</family>
<arch>x86_64</arch>
</os>
<property>
<name>fetch-native-hadoop</name>
</property>
</activation>
Comment on lines +2637 to +2646
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release manager may be using Mac or Linux, but needs to create package with lib for all supported platforms. I think these download/extract steps should be moved to existing dist profile (preferably in hadoop-ozone/dist/pom.xml).

Copy link
Copy Markdown
Contributor Author

@ChenSammi ChenSammi May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did put it into dist pom.xml in first version. But there are at least two more packages (unit test and integration test) requires native library to run(not covered in this patch). So to avoid duplicate the profiles in multiple pom.xml, gemini suggests to put it in root pom.xml, and with inherited false.

Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai May 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, if it's needed for tests, then keep it in root pom.xml, still, it should be platform-agnostic.

Copy link
Copy Markdown
Contributor Author

@ChenSammi ChenSammi May 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?

Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What "it should be platform-agnostic" specific mean, can you elaborate it a bit more?

Build "needs to create package with lib for all supported platforms" regardless of which OS/arch it runs on.

Copy link
Copy Markdown
Contributor Author

@ChenSammi ChenSammi May 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean? Or what you mean is we have two dist ozone-2.1.0-SNAPSHOT directory, for example ozone-2.1.0-SNAPSHOT-x86-64, and ozone-2.1.0-SNAPSHOT-aarch64?

I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.

Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both -Pnative and -Pnative-mac are enabled during maven build, the final package will have both linux native library and macos aarch64 library. Is this what you mean?

Yes.

I think -Pnative is the main one for official release, -Pnative-mac is mainly for dev with mac environment. I don't expect we will release macos package officially.

OK. (We do include some Mac/Windows native libs for Netty.)

Copy link
Copy Markdown
Contributor Author

@ChenSammi ChenSammi May 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

mvn clean package -DskipTests -Dmaven.javadoc.skip=true -s ~/workspace/tools/apache-maven-3.6.0/conf/settings-apache.xml -T 6 -Pdist,native,native-mac

sammi@X3HF9K66K7 target % ls -al ./ozone-2.1.0-SNAPSHOT/lib/native/
total 2056
drwxr-xr-x  6 sammi  staff     192 May 19 23:17 .
drwxr-xr-x  3 sammi  staff      96 May 19 23:17 ..
lrwxr-xr-x  1 sammi  staff      28 May 19 23:17 libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
lrwxr-xr-x  1 sammi  staff      25 May 19 23:17 libhadoop.so -> libhadoop_linux_x86_64.so
-rw-r--r--  1 sammi  staff  890256 May 19 23:17 libhadoop_linux_x86_64.so
-rw-r--r--  1 sammi  staff  157368 May 19 23:17 libhadoop_osx_aarch_64.dylib
sammi@X3HF9K66K7 target % tar -tvf ozone-2.1.0-SNAPSHOT.tar.gz| grep libhadoop
lrwxr-xr-x  0 sammi  staff        0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.dylib -> libhadoop_osx_aarch_64.dylib
-rw-r--r--  0 sammi  staff   157368 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_osx_aarch_64.dylib
-rw-r--r--  0 sammi  staff   890256 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop_linux_x86_64.so
lrwxr-xr-x  0 sammi  staff        0 May 19 23:17 ozone-2.1.0-SNAPSHOT/lib/native/libhadoop.so -> libhadoop_linux_x86_64.so

<build>
<plugins>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<version>${download-maven-plugin.version}</version>
<inherited>false</inherited>
<executions>
<execution>
<id>fetch-linux-library</id>
<goals>
<goal>wget</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<url>https://raw.githubusercontent.com/apache/ozone-thirdparty/master/hadoop-native-lib/${hadoop.version}/libhadoop_linux_x86_64.so</url>
<unpack>false</unpack>
<outputDirectory>${project.build.directory}/native-lib</outputDirectory>
<skipCache>false</skipCache>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>${maven-antrun-plugin.version}</version>
<inherited>false</inherited>
<executions>
<execution>
<id>extract-linux-library</id>
<goals>
<goal>run</goal>
</goals>
<phase>prepare-package</phase>
<configuration>
<target>
<exec dir="${project.build.directory}/native-lib" executable="bash" failonerror="true">
<arg value="-c" />
<arg value="ln -sf libhadoop_linux_x86_64.so libhadoop.so" />
</exec>
</target>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>native-mac</id>
<activation>
<os>
<family>mac</family>
<arch>aarch64</arch>
</os>
<property>
<name>fetch-native-hadoop</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<version>${download-maven-plugin.version}</version>
<inherited>false</inherited>
<executions>
<execution>
<id>fetch-mac-library</id>
<goals>
<goal>wget</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<url>https://raw.githubusercontent.com/apache/ozone-thirdparty/master/hadoop-native-lib/${hadoop.version}/libhadoop_osx_aarch_64.dylib</url>
<unpack>false</unpack>
<outputDirectory>${project.build.directory}/native-lib</outputDirectory>
<skipCache>false</skipCache>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>${maven-antrun-plugin.version}</version>
<inherited>false</inherited>
<executions>
<execution>
<id>extract-mac-library</id>
<goals>
<goal>run</goal>
</goals>
<phase>prepare-package</phase>
<configuration>
<target>
<exec dir="${project.build.directory}/native-lib" executable="bash" failonerror="true">
<arg value="-c" />
<arg value="ln -sf libhadoop_osx_aarch_64.dylib libhadoop.dylib" />
</exec>
</target>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Loading