[improve][build] Add Async Profiler integration for integration tests

[lhotari]

Motivation

Async Profiler is a powerful profiler, but it requires using Linux to get more accurate profiling results. When the main development workflow is on Mac for many like me, I started to wonder if it's possible to set up Async Profiler with the more accurate profiling mode using Docker. I use OrbStack (instead of Docker Desktop) on Mac to develop Apache Pulsar. I wanted to be able to create simple scenarios that take a few minutes to run and analyze the results with various tools. For example, converting flamegraphs to .html format with various views (such as per-thread CPU flamegraphs, allocation flamegraphs, and lock flamegraphs) and using Eclipse Mission Control to analyze GC events and other details that Async Profiler is able to collect together with Java Flight Recorder in its "jfrsync" mode.

This work extends the Async Profiler support that was added in #24623 for directly profiling unit tests in the same process. The downside of that approach is that there will be too many aspects included, which skew the results and make it not useful for validating the performance of actual production code. It also had the problem that Async Profiler on Mac doesn't support the more accurate CPU profiling modes that Async Profiler supports on Linux. This is explained in https://github.com/async-profiler/async-profiler/blob/master/docs/CpuSamplingEngines.md.

On Mac, it's not possible to profile native methods either, such as profiling the sources of exceptions. With Async Profiler, it's possible to specify event=Java_java_lang_Throwable_fillInStackTrace,tree,reverse to profile exception sources. (btw. JFR has also jdk.JavaExceptionThrow and jdk.JavaErrorThrow events which could be used, but the exception event isn't enabled by default).

Modifications

• Add changes to inttest Dockerfiles to install async-profiler
• Change build to support running and passing necessary parameters to integration tests for profiling
• Make changes to the PulsarContainer used in integration tests to integrate with Async Profiler and configure the Docker container with necessary privileges and security configuration
• Add a sample test class which profiles a broker while pulsar-perf consume and produce are running

Additional information

While profiling on Mac, I noticed that memory alloc and free would show up in flamegraphs. It seems that virtualization has an overhead and one way to reduce that is to use Transparent Huge Pages ("madvise" mode is recommended on Linux). That is explained in the sample test PulsarProfilingTest. That links to https://shipilev.net/jvm/anatomy-quarks/2-transparent-huge-pages/ which explains details. Large memory pages significantly reduce memory alloc and free overhead and this matters a lot in virtualized platforms. Therefore, it would also be useful to recommend -XX:+UseTransparentHugePages for Pulsar production deployments. -XX:+AlwaysPreTouch would also be recommended.
The throughput in Mac Orbstack improved over 10% and latency reduced even more for the sample PulsarProfilingTest that runs at full throughtput for 3 minutes.

For actual production environment profiling, besides async-profiler, AWS aperf seems like a great tool for Linux environments (it's not AWS specific). It also integrates with async-profiler.

AWS provides recommendations about using THP in "madvise" or "always" mode. ("always" mode could cause issues so I wouldn't recommend using it, my recommendation is "madvise" mode). There's also a recommendation about -XX:+UseTransparentHugePages

For Java, hugepages can be used for both the code-heap and data-heap by adding the below flags to your JVM command line

1. -XX:+UseTransparentHugePages when THP is set to at least madvise
2. -XX:+UseLargePages if you have pre-allocated huge pages through sysctl or the kernel command line.

and it also mentions:

Using huge-pages should generally improve performance on all EC2 instance types, but there can be cases where using exclusively huge-pages may lead to performance degradation. Therefore, it is always recommended to fully test your application after enabling and/or allocating huge-pages.

There's a significant high risk for performance degradation when using "always" mode (which is the Linux default). That risk is low for "madvise" mode. Some of the details are explained in this blog post: https://www.evanjones.ca/hugepages-are-a-good-idea.html .

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.11%. Comparing base (f767149) to head (6c58c8c).
⚠️ Report is 456 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #24688       +/-   ##
=============================================
+ Coverage     38.60%   74.11%   +35.50%     
- Complexity    12971    33005    +20034     
=============================================
  Files          1836     1893       +57     
  Lines        143700   147811     +4111     
  Branches      16616    17098      +482     
=============================================
+ Hits          55475   109545    +54070     
+ Misses        80664    29512    -51152     
- Partials       7561     8754     +1193     

Flag	Coverage Δ
inttests	26.20% <ø> (-0.50%)	⬇️
systests	22.67% <ø> (-0.02%)	⬇️
unittests	73.62% <ø> (+39.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1398 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.