Flaky-test: BlobStoreManagedLedgerOffloaderTest times out and the JVM runs out of memory

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/actions/runs/3050809339/jobs/4920269198#step:10:10631

Exception stacktrace

Full exception stacktrace

 Error:  Error occurred in starting fork, check output in log
  Error:  Process Exit Code: 3
  Error:  Crashed tests:
  Error:  org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest
  Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
  Error:  Command was /bin/sh -c cd /home/runner/work/pulsar/pulsar/tiered-storage/jcloud && /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+ExitOnOutOfMemoryError -Xmx1G -XX:+UseZGC -Dpulsar.allocator.pooled=true -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.out_of_memory_policy=FallbackToHeap -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED -jar /home/runner/work/pulsar/pulsar/tiered-storage/jcloud/target/surefire/surefirebooter1780857476976895968.jar /home/runner/work/pulsar/pulsar/tiered-storage/jcloud/target/surefire 2022-09-14T10-41-32_464-jvmRun1 surefire8895956485356214276tmp surefire_01993089739051770551tmp
  Error:  Error occurred in starting fork, check output in log
  Error:  Process Exit Code: 3
  Error:  Crashed tests:
  Error:  org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:511)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:383)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
  Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:247)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1161)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1002)
  Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:848)
  Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
  Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
  Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
  Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
  Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
  Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
  Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
  Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
  Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
  Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
  Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
  Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
  Error:  Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
  Error:  Command was /bin/sh -c cd /home/runner/work/pulsar/pulsar/tiered-storage/jcloud && /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+ExitOnOutOfMemoryError -Xmx1G -XX:+UseZGC -Dpulsar.allocator.pooled=true -Dpulsar.allocator.leak_detection=Advanced -Dpulsar.allocator.exit_on_oom=false -Dpulsar.allocator.out_of_memory_policy=FallbackToHeap -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED -jar /home/runner/work/pulsar/pulsar/tiered-storage/jcloud/target/surefire/surefirebooter1780857476976895968.jar /home/runner/work/pulsar/pulsar/tiered-storage/jcloud/target/surefire 2022-09-14T10-41-32_464-jvmRun1 surefire8895956485356214276tmp surefire_01993089739051770551tmp

# Created at 2022-09-14T10:42:34.673
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'java.lang.OutOfMemoryError: Java heap space'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'java.lang.OutOfMemoryError: Java heap space'.
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
	at java.base/java.lang.Thread.run(Thread.java:833)


# Created at 2022-09-14T10:42:34.675
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'Dumping heap to /tmp/java_pid64289.hprof ...'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'Dumping heap to /tmp/java_pid64289.hprof ...'.
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
	at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
	at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
	at java.base/java.lang.Thread.run(Thread.java:833)

heap dump: https://transfer.sh/NdL9hV/Unit-OTHER-heapdump.zip

test case execution logs: https://transfer.sh/z4KMtm/org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloaderTest-output.txt

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

over 500MB in the PulsarMockBookKeeper:

over 4 million LedgerEntryImpl instances

[lhotari]

This issue was fixed by #17743