[feat][ci] Collect code coverage for integration tests from docker containers

[lhotari]

Motivation

Code coverage metrics don't currently consider integration tests and systems tests that are run as part of the Pulsar CI pipeline. There hasn't been a way to measure the total code coverage of Pulsar CI pipeline tests although it would be useful to see what parts of Pulsar code base are tested and what aren't.

Modifications

• Add solution to Pulsar Testcontainer based integration tests so that the Jacoco agent gets activated for processes that are run inside Docker containers
  • A directory from the host is mounted in the docker container. This directory is used for storing the Jacoco exec files.
• The code coverage should be uploaded to codecov for all public code repositories. This is now changed so that it's possible to get coverage metrics for builds running in a fork.
• Graceful shutdown of Java processes is needed for collecting Jacoco exec data since by default, the Jacoco agent dumps the exec data to a file in a shutdown hook.
  • For processes managed with supervisor, it's necessary to run "supervisorctl stop all" to trigger a graceful shutdown
    • It was necessary to stop Zookeeper last for preventing shutdown getting stuck for Brokers or Bookies.
• Creating the Jacoco XML report needed for Codecov is created using Jacoco command line tools.
  • The script has been added as part of build/run_integration_group.sh script.
    • The jar files of the Docker image is needed to create the XML report from the exec files.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: lhotari#134

[lhotari]

@yaalsn Please review this PR.

[lhotari]

Example of code coverage results for an earlier experiment: https://app.codecov.io/github/lhotari/pulsar/commit/369dec198b5cf6b98cd5c52a0c32126974a8e9ac/tree

[lhotari]

Codecov results of the currently running PR build in the fork will be available at https://app.codecov.io/github/lhotari/pulsar/commit/23c1865976414a166e4ff2d6c5874937390cfc9e/tree

[lhotari]

#19264 will also be needed to fix coverage results.

[nicoloboschi]

This is awesome!

[lhotari]

/pulsarbot rerun-failure-checks

[aymkhalil]

I wonder if including integration tests in the code coverage report would hide gaps in unit test coverage (unless the latter coverage metric was already broken :/)

Is there a possibility to separate the coverage report for unit vs integ tests?

[lhotari]

I wonder if including integration tests in the code coverage report would hide gaps in unit test coverage (unless the latter coverage metric was already broken :/)

Is there a possibility to separate the coverage report for unit vs integ tests?

Yes, it should be possible to view unit test coverage separately. They are reported to Codecov with different flags.

[lhotari]

@aymkhalil The docs for the flags feature in Codecov: https://docs.codecov.com/docs/flags . It seems that the full feature is disabled by default.

[lhotari]

To experiment, I enabled it for my fork at https://app.codecov.io/github/lhotari/pulsar/flags . For apache/pulsar , I don't have access so I'll have to create a ASF ticket for handling that.