[improve][test] Improve integration test profiling test example

[lhotari]

Motivation & Modifications

A previous PR #24688 added an example test for profiling integration tests with Async Profiler.

Improvements:

• collecting more details from the pulsar-perf produce and pulsar-perf consume containers
  • store the latency histogram to the tests/integration/target directory
  • store stdout & stderr to a log file in the same directory
  • gracefully shutdown the containers so that the latency histogram files don't get truncated
• collect topic stats and internal stats in pretty printed JSON to stdout and to files stored in tests/integration/target
• collect broker metrics and store to tests/integration/target
  • metrics can be used to validate PIP-430 in this test so that all reads come from the cache in the tailing read scenario setup with pulsar-perf consume and pulsar perf produce
• switch to use smaller message size without batching to stress test PIP-430 changes
• configure dispatcherMaxReadBatchSize to 1000 in broker to get past issue [Bug] With small entry sizes and high throughput, it's necessary to increase dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes to utilize CPU on the broker #24695
  • this PR can be used to reduce issue [Bug] With small entry sizes and high throughput, it's necessary to increase dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes to utilize CPU on the broker #24695 by commenting out the setting
• improve integration test startup scripts
  • fix an issue with invalid replacements in conf/pulsar_env.sh
  • always add parameters to capture heap dumps on OOME
• install jq to test container images so that it can be used in pretty printing topic stats retrieved with curl

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.19%. Comparing base (669ab61) to head (89c82a9).
⚠️ Report is 454 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #24701   +/-   ##
=========================================
  Coverage     74.18%   74.19%           
+ Complexity    33472    33458   -14     
=========================================
  Files          1895     1895           
  Lines        147968   147968           
  Branches      17134    17134           
=========================================
+ Hits         109768   109780   +12     
+ Misses        29448    29420   -28     
- Partials       8752     8768   +16     

Flag	Coverage Δ
inttests	26.39% <ø> (+0.06%)	⬆️
systests	22.66% <ø> (-0.04%)	⬇️
unittests	73.69% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 71 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.