[Build/Tests] Make the test JVM exit if OOME occurs

[lhotari]

Motivation

• OOMEs can make the build to take very long to complete or to hang.
  It's better to fail fast in tests when OOMEs occur.

• example of a build failure where the build hangs: https://github.com/apache/pulsar/runs/5371747143?check_suite_focus=true

  • thread dump
  • it looks like Zookeeper client has gotten into a bad state because of OOME, however I'm not sure about this.

Modifications

• Add XX:+ExitOnOutOfMemoryError to test JVM arguments
• disable ClientDeduplicationFailureTest.testClientDeduplicationCorrectnessWithFailure completely since it leads to OOMEs in certain conditions.

[nicoloboschi]

+1, good idea!

[eolivelli]

what happens in case of OOM with this flag ?
is surefire reporting the name of the broker test ?
otherwise it will be very hard to understand why CI failed

[lhotari]

what happens in case of OOM with this flag ? is surefire reporting the name of the broker test ? otherwise it will be very hard to understand why CI failed

The OOME does get logged in surefire report files. IIRC, the error code was 3 for the terminated test JVM. This is visible in the console output.
It's actual even harder to understand why CI failed when the JVM keeps on running since a OOME can leave some components in a broken state.

[eolivelli]

let's try

+1

[michaeljmarshall]

Just created an issue to track the fact that we disabled a test: #14523.

[lhotari]

Here's an explanation and a fix for some of the recent OOMEs in tests: #14524 .

[poorbarcode]

Hi @lhotari

The 2.10 branch also occurs the OOM exception, so I cherry-picked this PR into the branch-2.10.

see: https://github.com/apache/pulsar/actions/runs/4566713860/jobs/8111430173?pr=19971