[feat][pip] PIP-478: Asynchronous v5 client auth plugin interfaces and TLS material provider plugin interface

[lhotari]

Motivation

The v4 client authentication surface has five concrete, observable problems, and they are what this
PIP addresses:

1. The Netty event loop blocks under credential refresh. ClientCnx.handleAuthChallenge() calls
   the plugin synchronously on the I/O thread, so an OAuth2 token-endpoint call or an Athenz ZTS
   query stalls every producer, consumer, and lookup multiplexed onto that loop.
2. AuthenticationDataProvider is a kitchen sink. 13 methods covering three unrelated concerns
   (TLS material, HTTP request auth, binary-protocol auth); every built-in implementation returns
   null or false from the methods it does not care about, and which transports a plugin actually
   supports is only discoverable at runtime.
3. Plugins cannot share the client's runtime. AuthenticationOAuth2 builds its own
   AsyncHttpClient (and its own SSL factory) because the SPI gives it nowhere to ask the client for
   one — issue #24795, with PR
   #24944 as the workaround.
4. The PIP-337 SSL factory is a kitchen sink coupled to the v4 auth SPI.
   PulsarSslConfiguration carries an AuthenticationDataProvider, and the factory interface
   exposes rebuild choreography (needsUpdate() / update() / createInternalSslContext() / …) as
   public API. It is also synchronous. Operationally this blocks the deployments that most need a
   plugin: those whose policy forbids storing TLS private keys in files at all — keys must stay in an
   HSM/KMS or be delivered in-memory by a workload-identity system (SPIFFE/SPIRE). The same
   configurability is also what makes a FIPS-compliant TLS transport practical.
5. Challenge-response auth is scattered across the stack. ClientCnx interleaves connect,
   refresh, and SASL rounds on one synchronous path, and on HTTP there is no generic multi-round
   driver at all — the SASL plugin implements the 401→resubmit→200 loop itself, and the admin
   (JAX-RS) client and the HTTP-lookup client each drive it separately, so the loop is effectively
   re-implemented per HTTP client API.

Related: PIP-466 (the v5 client API
this builds on) and PIP-337 (the SSL
factory this replaces).

Modifications

Adds the design document pip/pip-478.md. It builds on PIP-466's pulsar-client-api-v5 /
pulsar-client-v5 and adds two small, dependency-light SPI modules — pulsar-tls-factory-api and
pulsar-http-client-api — so the sibling broker-side PIP can depend on them without importing a
client artifact. In summary, three kitchen-sink surfaces are replaced by three focused SPIs:

• Async, capability-segregated v5 authentication SPI — a small core Authentication interface
  plus four narrow, opt-in capability interfaces, one per cell of a 2×2 matrix (binary vs HTTP
  transport × single-pass vs challenge-response). Every credential-producing method returns a
  CompletableFuture, so the event loop is never blocked, and challenge-response becomes an explicit
  capability driven generically by the framework on both the binary and HTTP paths.
• Two configuration paths, matching today's idioms: programmatic (authentication(myAuth), where
  the framework does not call configure(...)) and string-based (authPluginClassName +
  authParams, with configure(...) then initializeAsync(ctx) called once — the existing
  AuthenticationUtil.create(...) semantics).
• Framework-managed PulsarHttpClient SPI — the framework owns event loops, timers, DNS caches,
  and TLS refresh; plugins describe what they need via a PulsarHttpClientConfig and obtain an
  instance from AuthenticationInitContext.httpClientFactory() instead of building parallel
  infrastructure.
• Purpose-driven PulsarTlsFactory SPI, replacing PIP-337 — decoupled from the auth SPI,
  asynchronous, asking a factory only for a ready-built TLS object for a given purpose, and confining
  rotation to a reload callback (rebuild-not-mutate). Key material never crosses a Pulsar API, so
  HSM/KMS and workload-identity integrations become first-class instead of requiring a fork of the
  SSL layer.
• v5 client builder TLS configuration covering cert/key/trust material (PEM or keystore),
  ciphers/protocols, hostname verification, and three orthogonal provider axes: the TLS engine
  (JDK vs native OpenSSL), TlsPolicy.jsseProvider for the JSSE provider that builds the
  SSLContext and the key/trust manager factories, and TlsPolicy.jcaProvider for the JCA provider
  that parses and holds the key material (KeyStore, CertificateFactory, KeyFactory). Together
  with the engine selection already wired through the client and every server component, that makes a
  FIPS-compliant TLS transport configurable for FedRAMP/DoD/PCI-style deployments: BCJSSE (from
  bctls-fips) on the JDK engine, with BCFIPS (from bc-fips) as the crypto provider. Both axes
  are needed — BCJSSE registers no KeyStore service and BCFIPS no SSLContext, so pinning only the
  JSSE side leaves key material parsed outside the validated module. Deployments that must keep keys
  inside a PKCS#11 HSM use a custom PulsarTlsFactory instead.
• Compatibility bridge in both directions — LegacyV4AuthenticationAdapter runs an existing v4
  plugin as a v5 one with its blocking I/O forced onto ctx.blockingExecutor() (never the event
  loop), and V5ToV4AuthenticationAdapter lets a v5 plugin back the v4 client API.
• Internal migration of the Pulsar 5.0 client onto the v5 auth and TLS SPIs, including its v4
  client API implementation — which is how the event-loop fix reaches existing v4 applications with
  no source changes.

Breaking changes in 5.0 (detailed in the document): PIP-337 is removed outright — the
sslFactoryPlugin / sslFactoryPluginParams (and brokerClient*) config keys, the v4
ClientBuilder / PulsarAdminBuilder methods, the ClusterData fields, and the pulsar-admin clusters / pulsar-testclient CLI options — with tlsFactoryClassName / tlsFactoryConfig
successors, now on the v4 client and admin builders as well as server config, and a stale non-default
key rejected loudly at startup. TLS also becomes secure by default: hostname verification ON and
SAN-only matching (deprecated CN matching removed) — a documented breaking change, with remediation.

Out of scope: broker-side authentication (AuthenticationProvider / AuthenticationState),
deferred to a sibling PIP with the same design principles; removal of the v4 Authentication
interface and its deprecated methods, which are retained indefinitely for source compatibility (only
their bodies are re-implemented on top of the v5 SPI — the signatures stay); non-Java SDKs, each via
its own PIP; and the broader FIPS-compliance mode (approved algorithms in message encryption and
authentication, a FIPS distribution/packaging variant, a fail-fast fipsMode switch) — this PIP
covers only the TLS-transport slice.

Known gap: the proxy's own broker-client credential I/O still runs inline on the proxy's Netty
loop, because its lookup path uses a bare ConnectionPool and its data path a hand-rolled
DirectProxyHandler, neither of which binds the client auth services — the same behavior as v4, not
a regression, and deferred to a follow-up.

This is a documentation-only PR (the design proposal); implementation lands in separate PRs.

Links

Rendered PIP document:
https://github.com/lhotari/pulsar/blob/lh-pip-478-v5-client-async-auth-and-tls-material-provider/pip/pip-478.md

Discuss thread:
https://lists.apache.org/thread/s9n9jksr9vqgn9o982zmnnkcxdcncy3f

Vote thread:
https://lists.apache.org/thread/6n6q014236l4tstrxn9t4jbrp8xty997

[lhotari]

The PIP will be simplified (the PulsarTlsMaterialProvider part). I haven't yet updated the PIP document.

[lhotari]

The vote has passed and this PR PIP is waiting for a review.