[fix][proxy] Refresh auth data if ProxyLookupRequests

[michaeljmarshall]

Fixes: #10816
PIP: #19771
Supersedes: #19026
Depends on: #20062

Motivation

The Pulsar Proxy does not properly handle authentication data refresh when in state ProxyLookupRequests. The consequence is described in #10816. Essentially, the problem is that the proxy caches stale authentication data and sends it to the broker leading to connection failures.

#17831 attempted to fix the underlying problem, but it missed an important edge cases. Specifically, it missed the case that the ConnectionPool will have multiple connections when a lookup gets redirected. As such, the following problem exists (and is fixed by this PR):

1. Client opens connection to perform lookups.
2. Proxy connects to broker 1 to get the topic ownership info.
3. Time passes.
4. Client does an additional lookup, and this topic is on a newly created broker 2. In this case, the proxy opens a new connection with the stale client auth data.
5. Broker 2 rejects the connection because it fails with expired authentication.

Modifications

• Remove some of the implementation from [fix][proxy] Fix refresh client auth #17831. This new implementation still allows a broker to challenge the client through the proxy, but notably, it limits the number of challenges sent to the client. Further, the proxy does not challenge the client when the auth data is not expired.
• Introduce authentication refresh in the proxy so that the proxy challenges the client any time the auth data is expired.
• Update the ProxyClientCnx to get the clientAuthData from the ProxyConnection to ensure that it gets new authentication data.
• Add clock skew to the AuthenticationProviderToken. This is necessary to make some of the testing not flaky and it will also be necessary for users to configure in their clusters.

Verifying this change

The ProxyRefreshAuthTest covers the existing behavior and I expanded it to cover the edge case described above.

Additionally, testing this part of the code will be much easier to test once we implement #19624.

Documentation

• doc-not-needed

Matching PR in forked repository

PR in forked repository: the relevant tests pass locally, so I am going to skip the forked tests.

[lhotari]

LGTM

[nodece]

LGTM

[michaeljmarshall]

@cbornet - I think this is a necessary bug fix for 3.0. Without it, OIDC will not work correctly through the pulsar proxy. This bug was not introduced recently, but it didn't matter until now because the OIDC work exposes the bug. I've been fighting with CI all day due to flaky tests (and some legitimate failures, too). If we're able to hold off on creating branch-3.0 just a little longer, I'd really appreciate it. Thanks!