[feat][ci] Add integration to ge.apache.org Gradle Enterprise server

[lhotari]

Motivation

ASF Infra team together with Gradle Inc. has setup a dedicated Gradle Enterprise instance for Apache projects at https://ge.apache.org/ .

Gradle Enterprise works for both Gradle builds and Maven builds. There's a Maven extension which provides the integration for Maven builds, https://docs.gradle.com/enterprise/maven-extension/ .

You can find more information about the benefits of Gradle Enterprise on https://gradle.com . Gradle Enterprise is also for Maven builds!

Here's an example of builds scans for all Maven builds that are run as part of Pulsar CI workflow: https://ge.apache.org/scans?search.names=Git%20commit%20id%20short&search.timeZoneId=Europe/Helsinki&search.values=db7ab997&selection.buildScanB=ohtx53k6ymsgc
There are 69 maven builds that are run in total.

The build scans are also linked from the GitHub Actions results page, for example https://github.com/apache/pulsar/actions/runs/3839199234#summary-10438505199 shows this.

Mailing list discussion: https://lists.apache.org/thread/d36xbj8b7ntcyp85lv0730xfj4hodqxj

Modifications

Add configuration for Gradle Enterprise which will be activated only for builds in branches on apache/pulsar when a secret called GE_ACCESS_TOKEN exists.
A new environment variable called JOB_NAME is added so that the name of the GitHub Actions CI job can be passed on to Gradle Enterprise build scan values. GitHub Actions has a limitation that the job name isn't available as an environment variable.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.73%. Comparing base (9ef54fd) to head (e4ad807).
⚠️ Report is 3025 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19133      +/-   ##
============================================
- Coverage     47.22%   46.73%   -0.50%     
+ Complexity    10713    10344     -369     
============================================
  Files           713      687      -26     
  Lines         69697    67452    -2245     
  Branches       7485     7240     -245     
============================================
- Hits          32914    31521    -1393     
+ Misses        33096    32390     -706     
+ Partials       3687     3541     -146     

Flag	Coverage Δ
unittests	46.73% <ø> (-0.50%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 66 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codelipenghui]

/pulsarbot run-failure-checks

[tisonkun]

@lhotari Thanks for your contribution!

It should be good to try. I have some questions here:

1. How do you imagine we make use of the scan report?
2. How much time does the new step consume?
3. Is it possible to replace codecov with this scan so we reduce too many reports?

[tisonkun]

Also, maybe we can tag ready-for-test on this specific PR to preview the result now? I don't know exactly whether it works on a personal fork. Since it depends on ge.apache.org, I guess no?

[lhotari]

@lhotari Thanks for your contribution!

It should be good to try. I have some questions here:

1. How do you imagine we make use of the scan report?

there are many ways to make use of the scan reports. For example, the test execution reports are useful for analysing flaky tests and slow tests. Here's an example of test execution reports:
https://ge.apache.org/scans/tests?search.rootProjectNames=Pulsar&search.timeZoneId=Europe/Helsinki&tests.container=*
It seems that @poorbarcode has been working on improvements for slow tests in #17657 and child issues. Gradle Enterprise test reporting is a good way to find the slow tests.

For other parts of the build, the build scan contains detailed break downs of where time is spent in the build. It's simply a way to get a better understanding of what happens during the build. One of the challenges is that Gradle Enterprise doesn't seem to provide a nice way to group the related builds in list views. There are links in the build scan report header to find other builds in the same workflow run or for the same git commit hash.

However this isn't very helpful in the list views since there's no information there to see what build it actually is. That's why I added the links to the reports to the GitHub Actions workflow run summary view:

2. How much time does the new step consume?

The Gradle Enterprise maven extension collects the information and logs and sends them when the build finishes. It doesn't seem to add more than a few seconds to execution. One detail is that this solution is only enabled for builds that happen on the apache/pulsar branches. The reason for this is that Gradle Enterprise has a secret token that is used for publishing to ge.apache.org . If we'd like to publish reports for pull requests, the token would have to be made public. I don't think that there are plans for doing that at the moment.

3. Is it possible to replace codecov with this scan so we reduce too many reports?

This isn't a replacement for codecov. Currently codecov integration in apache/pulsar seems to be invalid and the results don't make sense. I think we should disable codecov reporting until it is fixed and it provides consistent results.

[lhotari]

Also, maybe we can tag ready-for-test on this specific PR to preview the result now? I don't know exactly whether it works on a personal fork. Since it depends on ge.apache.org, I guess no?

@tisonkun I created lhotari#123 to ensure that the changes don't break builds in forks. You won't be able to publish build scans to ge.apache.org since the secret token is only available for builds on branches that exist in apache/pulsar repository. As a solution for forked builds, I think we could add a way to use public Gradle builds scans available at https://scans.gradle.com/ .

Currently, the maven extension will only get activated when the token for Gradle Enterprise is available. The activation happens in .github/actions/gradle-enterprise/action.yml by copying .mvn/ge-extensions.xml to .mvn/extensions.xml and by making the token available in GRADLE_ENTERPRISE_ACCESS_KEY environment variable that the Gradle Enterprise maven extension uses.

[lhotari]

There seems to be some compatibility issue with Gradle Enterprise maven extension and the way how we retry tests with TestNG.

There's an error
Received a start event for 'DefaultTestDescriptor{id=-6948655823765472571, surefireForkId=2, name='org.apache.pulsar.client.api.SimpleSchemaTest', className='org.apache.pulsar.client.api.SimpleSchemaTest', parentId=3213794952218547303}' with duplicate id '-6948655823765472571'.

in build scan: https://ge.apache.org/s/lfbhlzzwce72c/failure?focused-exception-line=0-5#1

in GitHub Actions logs: https://github.com/apache/pulsar/actions/runs/3843180558/jobs/6546220728#step:11:1408

[lhotari]

I have isolated the issue in https://github.com/lhotari/gradle-enterprise-duplicate-test-id-repro

[lhotari]

I have isolated the issue in https://github.com/lhotari/gradle-enterprise-duplicate-test-id-repro

There's now a workaround in place to handle the issue. There was only one occurrence of the issue in the Pulsar code base.

[tisonkun]

Generally looks good. Let's see what it brings to us.

BTW, compatibility issues can be a risk that there're further unstable cases so we need to investigate it.

[tisonkun]

Please file an issue and add a reference for this trick.

[nicoloboschi]

LGTM

[lhotari]

BTW, compatibility issues can be a risk that there're further unstable cases so we need to investigate it.

@tisonkun I have reported this issue https://github.com/lhotari/gradle-enterprise-duplicate-test-id-repro to Gradle support. They will work on a fix eventually. One of the workarounds that was suggested, was switching to use JUnit Jupiter to run tests with testng-engine. (the problem doesn't appear with testng-engine) This is probably something we would like to do if we would like to migrate to Junit 5 eventually. testng-engine doesn't support testng xml test suites that are used by Pulsar integration tests.