[BUG] Making EventHub/ServiceBus IT more stable

[fangjian0423]

Context

Pipeline IT jobs always run failed, refer Jobs in run #20221004.1, Jobs in run #20221007.1 and Jobs in run #20220929.1.

It seems both IT failed because of the Access control is not active.

[com.azure.storage](http://com.azure.storage/).blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match' ..

reactor.core.Exceptions$ErrorCallbackNotImplemented: [com.azure](http://com.azure/).messaging.servicebus.ServiceBusException: Unauthorized access. 'Send' claim(s) are required to perform this operation. Resource: 'sb://***.***/topic1'. TrackingId:3ff7c86d4b574b55b8a2f5ab7653f5ff_G26, SystemTracker:gateway7, Timestamp:2022-10-04T15:28:18

com.azure.messaging.servicebus.ServiceBusException: Unauthorized access. 'Listen' claim(s) are required to perform this operation. Resource: 'sb://***.***/topic1/subscriptions/topicsub'. TrackingId:b416725b32a64130b324c61e81f86f4f_G28, SystemTracker:gateway7, Timestamp:2022-09-29T07:24:16, errorContext[NAMESPACE: ***.***. ERROR CONTEXT: N/A, PATH: topic1/subscriptions/topicSub, REFERENCE_ID: topic1/subscriptions/topicSub_df63ff_1664436255091, LINK_CREDIT: 1]

Rerun IT maybe success but I don't think it is the best way. We need investigate why it always run failed.

Goal

Making java-spring-test pipeline jobs more stable.

[fangjian0423]

IT added produce error scenario before, #31109

[fangjian0423]

Access control scenarios

1. Event Hubs: Storage ✅, Data ❌

Message produce failed. We can get error message in error MessageChannel and add retry configuration items.

2. Event Hubs: Storage ❌, Data ✅

Message produce success, consume error, and we can't get error message in error MessageChannel.

Message consume will auto retry by ScheduledExecutorService. Besides, we should modify partition start position(other wise, we can't receive the message produce before).

3. Event Hubs: Storage ❌, Data ❌

Similar with scenario 1 and 2.

4. Event Hubs: Storage ✅, Data ✅

Event Hubs cases run successfully.

5. Service Bus: Data ❌

Similar with scenario 1. We can add retry configuration items.

6. Service Bus: Data ✅

Service Bus cases run successfully.

Conclusion

1. Adding retry configuration items in message produce phase.
2. Modify partition start position to earliest in message consume phase.
3. CountDownLatch await time of all cases update to 600s.

[fangjian0423]

Josh confirm that there's no lag for the permission to take effect.

Besides, Bill takes a look at the the pipeline log. He found the token acquisition succeeded, but it's really strange the permissions issue occur.

And he gives some advices: some part of the environment isn’t being cleaned up and thus being configured by some other test resource deployment.

[fangjian0423]

The assignment role is correct by run log.

Besides, the test resource will be removed at last by run log.

Other useful information: IT runs begin at 15:27:15 PM, and the test resource deploy successfully at 15:24:10 PM.

[fangjian0423]

Now the permissions error occur occasionally.

After team discussion, we decide to keep watching the pipeline. if the permissions error occur frequently, we will try to fix it.