[Bug] ZKSessionWatcher will trigger a ConnectionLost session event if query takes more than 1/15 of zk session time, causing flushing of all caches

[lhotari]

@gaozhangmin During a test related PR #24171 I noticed this about the same code location:

pulsar/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKSessionWatcher.java

Lines 108 to 115 in 6e8c349

	try {
	zkClientState = future.get(tickTimeMillis, TimeUnit.MILLISECONDS);
	} catch (TimeoutException e) {
	// Consider zk disconnection if zk operation takes more than TICK_TIME
	zkClientState = Watcher.Event.KeeperState.Disconnected;
	}
	checkState(zkClientState);

This will trigger a ConnectionLost session event. That's why a delay should be kept under 2000 ms.
Currently MockZooKeeper's session timeout is hard coded to 30000ms and the check interval is 1/15 of this, therefore 2000ms.

When there's a test that blocks the ZooKeeper thread for more than 2000ms, it will result in a ConnectionLost event.
This PR makes the session timeout configurable and it's possible to increase the session timeout for tests where it's necessary to use a longer session timeout to avoid ConnectionLost session events.

This applies likewise to production code. This is problematic since the effective timeout becomes 1/15 of the zk session timeout for Pulsar operations.

Together with this change made in #23018 it will add significant load to ZooKeepers:

pulsar/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java

Lines 513 to 519 in 829df71

	// Clear cache after session expired.
	if (event == SessionEvent.SessionReestablished || event == SessionEvent.Reconnected) {
	for (MetadataCacheImpl metadataCache : metadataCaches) {
	metadataCache.invalidateAll();
	}
	invalidateAll();
	}

I'll file a new issue about resolving this problem.

Originally posted by @lhotari in #24624 (comment)

[lhotari]

@merlimat any suggestions about how to solve this issue?