[BUG] ManagementChannel throw exceptions after client closing

[wedmeed]

Describe the bug
I use EventProcessorClient for receiving EventHub events. The client has a "stop" method. After closing (300 seconds) I see the following errors:

Exception or Stack Trace

2020-09-09 16:56:30.181  WARN 94215 --- [       single-1] a.i.RequestResponseChannel<mgmt-session> : Retry #1. Transient error occurred. Retrying after 4511 ms.
The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:aa5d4bbb5a634dacb9f6c603ee2d7db8_G17, SystemTracker:gateway5, Timestamp:2020-09-09T12:56:30, errorContext[NAMESPACE: xxx.servicebus.windows.net, PATH: $management, REFERENCE_ID: mgmt:sender, LINK_CREDIT: 69]
2020-09-09 16:56:30.181 ERROR 94215 --- [       single-1] c.a.m.e.i.ManagementChannel              : Exception occurred:
The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:aa5d4bbb5a634dacb9f6c603ee2d7db8_G17, SystemTracker:gateway5, Timestamp:2020-09-09T12:56:30, errorContext[NAMESPACE: xxx.servicebus.windows.net, PATH: $management, REFERENCE_ID: mgmt:sender, LINK_CREDIT: 69]
2020-09-09 16:56:30.181 ERROR 94215 --- [       single-1] c.a.c.a.i.RequestResponseChannel<mgmt>   : Exception in RequestResponse links. Disposing and clearing unconfirmed sends.
The connection was inactive for more than the allowed 300000 milliseconds and is closed by container 'LinkTracker'. TrackingId:aa5d4bbb5a634dacb9f6c603ee2d7db8_G17, SystemTracker:gateway5, Timestamp:2020-09-09T12:56:30, errorContext[NAMESPACE: xxx.servicebus.windows.net, PATH: $management, REFERENCE_ID: mgmt:sender, LINK_CREDIT: 69]

To Reproduce

1. Create a client

EventProcessorClientBuilder eventProcessorClientBuilder = new EventProcessorClientBuilder()
                    .connectionString(
                            properties.getConnectionString(),
                            properties.getConsumers().get(hubName).getDestination())
                    .consumerGroup(consumer.getGroup())
                    .processEvent(this::onReceive)
                    .processError(this::onError)
                    .checkpointStore(new BlobCheckpointStore(blobContainerAsyncClient))
                    .processPartitionInitialization(this::onInitialize)
                    .processPartitionClose(this::onClose)
                    .trackLastEnqueuedEventProperties(true)
                    .initialPartitionEventPosition(positionMap);

            internalClient = eventProcessorClientBuilder.buildEventProcessorClient();

2. Wait for system started to receive from some partitions
3. Stop receiving by

internalClient.stop();

Expected behavior
All channels are closed gracefully without exceptions and error messages.

Setup (please complete the following information):

• OS: MacOS 10.14
• IDE : IntelliJ IDEA
• Version of the Library used:
  implementation 'com.azure:azure-messaging-eventhubs:5.1.2'
  implementation 'com.azure:azure-messaging-eventhubs-checkpointstore-blob:1.1.2'

[joshfree]

Thanks for reporting this issue @wedmeed. @YijunXieMS can you please follow up?

[joshfree]

Linking related issue #15100

[samuelkoppes]

We are investigating this issue and will report back with an update by 11/4/2020.