Skip to content

[fix][test] Reduce admin client churn in ExtensibleLoadManagerTest.startBroker - #25676

Merged
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-flaky-extensible-load-manager-start-broker
May 5, 2026
Merged

[fix][test] Reduce admin client churn in ExtensibleLoadManagerTest.startBroker#25676
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-flaky-extensible-load-manager-start-broker

Conversation

@merlimat

@merlimat merlimat commented May 5, 2026

Copy link
Copy Markdown
Contributor

Motivation

ExtensibleLoadManagerTest.startBroker is a @BeforeMethod that runs before each test method (e.g. testStopBroker, testIsolationPolicy) to make sure all brokers are running and ready. Some test methods intentionally stop brokers, so the next method's startBroker waits for those brokers to come back.

The readiness check polled by creating a brand new PulsarAdmin per broker on every poll iteration:

for (BrokerContainer brokerContainer : pulsarCluster.getBrokers()) {
    try (PulsarAdmin brokerAdmin = PulsarAdmin.builder()
            .serviceHttpUrl(brokerContainer.getHttpServiceUrl()).build()) {
        ...
    }
}

That's 3 brokers × N polls of admin builder/close, each one establishing a new HTTP client. The connection setup work piles up against brokers that are still warming up after a stop/restart and the await can exhaust its 180s budget.

Example failure: https://scans.gradle.com/s/4dgsnjrtxayqa/tests/task/:tests:integration:integrationTest/details/org.apache.pulsar.tests.integration.loadbalance.ExtensibleLoadManagerTest/startBroker?top-execution=1

Modifications

Build the per-broker PulsarAdmin clients once before the await loop and reuse them across all poll iterations. Close them in a finally block. The readiness checks themselves (getActiveBrokers, createPartitionedTopic, lookupPartitionedTopic) are unchanged.

Verifying this change

This change is already covered by the existing ExtensibleLoadManagerTest integration test (which runs startBroker between every test method).

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…artBroker

The @BeforeMethod startBroker() polled readiness by creating a new
PulsarAdmin instance per broker on every poll iteration (3 brokers x N
polls of build/close). When earlier tests stopped brokers
(testStopBroker, testIsolationPolicy), the brokers in the next method's
startBroker() are still warming up, and the per-tick connection setup
contends with that warmup.

Build the per-broker PulsarAdmin clients once before the await loop and
reuse them across all poll iterations, closing them in a finally block.
The readiness checks themselves (getActiveBrokers / createPartitioned-
Topic / lookupPartitionedTopic) are unchanged.
@merlimat
merlimat merged commit 272c239 into apache:master May 5, 2026
80 of 82 checks passed
poorbarcode pushed a commit to poorbarcode/pulsar that referenced this pull request May 6, 2026
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants