Admin create subscription intermittently fails in multi-broker config

[cdbartholomew]

Describe the bug
When I try to create a subscription using pulsar-admin it intermittently fails with 504 Gateway Timeout. If I try again later, the creation will eventually succeed. I see the following exception in the broker log when it fails:

17:38:07.080 [pulsar-web-28-28] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null][persistent://public/useast1-gcp/test5] Creating subscription testsub3 at message id 9223372036854775807:9223372036854775807:-1 17:38:07.081 [pulsar-web-28-28] WARN org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] [persistent://public/useast1-gcp/test5] Failed to create subscription testsub3 at message id 9223372036854775807:9223372036854775807:-1 javax.ws.rs.WebApplicationException: HTTP 307 Temporary Redirect at org.apache.pulsar.broker.web.PulsarWebResource.validateTopicOwnership(PulsarWebResource.java:557) ~[org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.validateAdminOperationOnTopic(PersistentTopicsBase.java:253) ~[org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalCreateSubscription(PersistentTopicsBase.java:904) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.pulsar.broker.admin.v2.PersistentTopics.createSubscription(PersistentTopics.java:376) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]

This is similar to #1615, which is closed.

To Reproduce
Steps to reproduce the behavior:

1. Set up a multi-broker cluster.
2. Try to add a subscription to an existing topic.

pulsar-admin topics create-subscription -s testsub3 persistent://public/useast1-gcp/test5

3. Repeat step 2 with new subscriptions until the command fails

Expected behavior
Subscription creation should always succeed. If it is going to fail, it shouldn't just time out, but return a meaningful error code/message.

Screen capture
Here are the commands I ran:
$ pulsar-admin topics create-subscription -s testsub3 persistent://public/useast1-gcp/test5
HTTP 504 Gateway Timeout

Reason: HTTP 504 Gateway Timeout
$ pulsar-admin topics create-subscription -s testsub3 persistent://public/useast1-gcp/test5

This second attempt succeeded.

Additional context
I am running 2.2.1 in k8s. I have 3 brokers and 3 proxies.

[cdbartholomew]

This is working in 2.4.2. It was fixed under #5360. Closing.