Skip to content

[fix][test] Fix flaky ExtensibleLoadManagerTest.startBroker timeout - #25500

Merged
nodece merged 1 commit into
apache:masterfrom
lhotari:lh-fix-flaky-ExtensibleLoadManagerTest-startBroker
Apr 9, 2026
Merged

[fix][test] Fix flaky ExtensibleLoadManagerTest.startBroker timeout#25500
nodece merged 1 commit into
apache:masterfrom
lhotari:lh-fix-flaky-ExtensibleLoadManagerTest-startBroker

Conversation

@lhotari

@lhotari lhotari commented Apr 9, 2026

Copy link
Copy Markdown
Member

Fixes #21875

Motivation

Fix flaky ExtensibleLoadManagerTest.startBroker that times out when restarting brokers after testStopBroker or testIsolationPolicy.

The root cause is that after a broker is stopped and its container is restarted, the ServiceUnitStateTableViewImpl channel fails to start due to a compacted ledger read error ("Invalid unknown tag type: 4", a known issue per PIP-429). The broker process crashes and restarts 2-3 times inside the Docker container before succeeding. The previous 120-second timeout was insufficient for this crash-restart cycle.

Additionally, ignoreExceptions() was hiding all errors, making these failures impossible to debug from test logs.

Modifications

  • Replace ignoreExceptions() with explicit try-catch and logging inside the Awaitility condition, so errors are visible in test logs while still allowing retries
  • Add informational logging when brokers don't see the expected number of active brokers
  • Increase the Awaitility timeout from 120 to 180 seconds to accommodate broker crash-restart cycles
  • Minor: early return when pulsarCluster is null, rename shadowed variable

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing integration tests in the pulsar-loadbalance test group.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Replace ignoreExceptions() with explicit try-catch and logging in the
startBroker @BeforeMethod so that errors are visible in test logs while
still allowing retries. Increase the Awaitility timeout from 120 to 180
seconds to accommodate broker crash-restart cycles that occur when the
ServiceUnitStateTableViewImpl fails to read the compacted ledger.
@nodece
nodece merged commit 12bcd6b into apache:master Apr 9, 2026
80 of 82 checks passed
@lhotari lhotari added this to the 4.3.0 milestone Apr 21, 2026
lhotari added a commit that referenced this pull request Apr 21, 2026
lhotari added a commit that referenced this pull request Apr 21, 2026
lhotari added a commit that referenced this pull request Apr 21, 2026
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 23, 2026
nodece pushed a commit to ascentstream/pulsar that referenced this pull request May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky-test: ExtensibleLoadManagerTest.startBroker

2 participants