[improve][build] Support building docker images that use Wolfi base image which is glibc based

[lhotari]

Motivation

Mailing list thread: https://lists.apache.org/thread/281kwr6txg6zgtqx5lry0fcox66gky6n

One of the drawbacks of the Alpine based image is that it's based on the "musl" libc library which causes incompatibilities for binaries that are compiled for glibc library. In Pulsar, we have already resolved this for binaries distributed with Pulsar and Pulsar IO connectors.

However, there's another disadvantage. The memory allocation library (malloc) in Alpine doesn't contain any tuneables and in Alpine, there's no documentation how to tune malloc operations.

It's possible to replace the default malloc library with other allocation libraries, but this could cause other instability.

For benchmarking purposes, it would be useful to be able to build a glibc based Docker image for Pulsar and specify glibc memory allocation tuneables.

Various glibc tuneables can be viewed with this command:

docker run --rm -it cgr.dev/chainguard/wolfi-base sh -c "/lib64/ld-linux-*.so.1 --list-tunables"

The reference: https://sourceware.org/glibc/manual/latest/html_node/Memory-Allocation-Tunables.html

For example, to enable Transparent Huge Pages (THP) for malloc allocations:

export GLIBC_TUNABLES=glibc.malloc.hugetlb=1

Great resources about Transparent Huge Pages in Linux:

• https://www.oxla.com/blog/using-huge-pages-in-linux-applications
• https://www.oxla.com/blog/using-huge-pages-in-linux-applications-part-2-transparent-hugepage
• https://kernel.org/doc/html/latest/admin-guide/mm/transhuge.html

Alpine musl malloc criticism:

• https://nickb.dev/blog/default-musl-allocator-considered-harmful-to-performance/
  • HN discussion

For performance testing, it might be useful to also set glibc.malloc.mmap_threshold to the bytes value of 2MB:

export GLIBC_TUNABLES=glibc.malloc.hugetlb=1:glibc.malloc.mmap_threshold=2097152

For k8s or Docker configurations GLIBC_TUNABLES would be passed with the value glibc.malloc.hugetlb=1:glibc.malloc.mmap_threshold=2097152.

The reason why this matters is that most of BookKeeper's and Pulsar's memory is allocated by Netty. Under the covers the allocations are made using os::malloc which maps to the glibc malloc. To get the performance benefits of Transparent Huge Pages, it's necessary that the allocations to the operating system are made using mmap with madvise for MADV_HUGEPAGE. The Java VM will handle this for the Java heap memory with -XX:+TransparentHugePages, but that's not sufficient, since it doesn't cover the allocations made by Netty and RocksDB. When the malloc library has support for THP, there are better ways to control the behavior. When using glibc, it will mark the large allocations with MADV_HUGEPAGE when glibc.malloc.hugetlb=1 is used and closes that gap for Pulsar.

THP is also supported in many other allocators and for example Microsoft's mimalloc (https://github.com/microsoft/mimalloc) could be another option. However, I faced some stability issues with it on Alpine. It's possibly an issue when LD_PRELOAD contains both gcompat and mimalloc libraries. In the Alpine images, gcompat is required for supporting Netty native libraries which have been built for glibc. This compatibility issue is also solved by using the Wolfi image.

Additional Information

See #24688 regarding profiling.

Modifications

• add additional Dockerfiles Dockerfile.wolfi for pulsar and pulsar-all images.
  • Wolfi (cgr.dev/chainguard/wolfi-base) is used as the base image
  • More information about Wolfi OS: https://github.com/wolfi-dev/
    • Wolfi uses apk for package management like Alpine and package names are similar
    • Wolfi is backed by Chainguard and they release security updates to CVEs nightly
• make changes to the maven build that Wolfi based images will be built when -Pdocker-wolfi is passed to the command line
• add GitHub Actions workflow changes to build the Wolfi images when Dockerfile.wolfi changes
• add GitHub Actions workflow changes to support building Wolfi images and running integration and system tests with the Wolfi images when the workflow gets triggered manually (example build)

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.23%. Comparing base (f1b66ae) to head (01da5f1).
⚠️ Report is 456 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24692      +/-   ##
============================================
+ Coverage     74.21%   74.23%   +0.01%     
+ Complexity    33463    33108     -355     
============================================
  Files          1895     1895              
  Lines        147954   147954              
  Branches      17130    17130              
============================================
+ Hits         109805   109831      +26     
+ Misses        29387    29372      -15     
+ Partials       8762     8751      -11     

Flag	Coverage Δ
inttests	26.30% <ø> (+0.01%)	⬆️
systests	22.66% <ø> (-0.03%)	⬇️
unittests	73.74% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 81 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nodece]

Overall, it looks good to me. Could you add a job to build the Wolfi base image in the GHA? It would be great to verify that as well.

[lhotari]

Overall, it looks good to me. Could you add a job to build the Wolfi base image in the GHA? It would be great to verify that as well.

@nodece I pushed changes to build the Wolfi images when there are file changes directly related to Dockerfile.wolfi files so that our CI doesn't become slower. There's also the choice to manually trigger a build that uses the Wolfi base images. PTAL