Add HTTP/2 PING for broken connection detection.

[jeet1995]

What

Adds an HTTP/2 PING keepalive handler that sends periodic PING frames on idle H2 parent channels to:

1. Prevent middlebox connection reaping -- NAT gateways, firewalls, and load balancers silently drop idle connections after vendor-specific timeouts (typically 4-5 min). The client then sees opaque TCP RST or timeout errors.
2. Detect broken connections -- if no PING ACK within the configured timeout after consecutive retries, the handler closes the connection so the pool creates a fresh replacement.

Timing parameters: PING interval=1s, ACK timeout=2s, consecutive-failure threshold=5 (worst-case eviction ≈ 12–15s). All three are tunable via COSMOS.HTTP2_PING_* system properties / env vars. The choice of threshold=5 is intentional — it trades a few extra seconds of detection latency for resilience against transient blips. Note: Rust's consecutive_failure_threshold is a per-HTTP-request shard-health counter (not a per-PING-ACK count) and is not directly comparable.

Why not reactor-netty's native pingAckTimeout / pingAckDropThreshold?

Reactor-netty 1.2.12+ exposes pingAckTimeout(Duration) and pingAckDropThreshold(int). These have a fundamentally different purpose and timing model:

Different definition of "idle"

From the reactor-netty docs:

"When a connection becomes idle (no reads/writes for the configured idle timeout duration), a PING frame is sent to check if the peer is still responsive."

"Important: This setting only takes effect when used together with ConnectionProvider with maxIdleTime configured. Without an idle timeout, PING frames will not be sent."

This is a "verify before evict" mechanism -- it sends a PING only when the full maxIdleTime has elapsed (e.g. 60s), as a last-ditch check before pool eviction:

reactor-netty native (maxIdleTime=60s, pingAckTimeout=10s):
t=0s   last activity
t=60s  idle timeout reached -> send PING
t=70s  no ACK -> close
Total detection time: 60-70 seconds

Our handler: continuous proactive keepalive

Our Http2PingHandler sends PINGs continuously at a fixed interval (default 1s) while idle, regardless of maxIdleTime:

1. Keepalive: each PING resets middlebox idle timers, preventing silent reaping
2. Fast detection: consecutive failures trigger close -- worst case ~15s (5 rounds x 3s)

Custom Http2PingHandler (interval=1s, timeout=2s, threshold=5):
t=0s   last activity
t=1s   PING #1 -> ACK (keepalive)
t=2s   PING #2 -> ACK (keepalive)
...
t=N    PING -> no ACK (connection broken)
t=N+2  timeout (failure 1/5) -> retry
...
t=N+12 timeout (failure 5/5) -> close

Summary

Aspect	reactor-netty pingAckTimeout	Custom Http2PingHandler
Purpose	Verify before idle eviction	Continuous keepalive + failure detection
When PINGs sent	Once, after full maxIdleTime (e.g. 60s)	Every 1s while idle
Requires maxIdleTime	Yes -- no PINGs without it	No -- independent
Detection time	60-70s	~15s (5 consecutive failures)
Middlebox keepalive	No -- already idle 60s before first PING	Yes -- 1s interval prevents reaping
Idle-based gating	N/A	Skips PING when last inbound activity was within the interval

Design decisions

1. Parent channel installation -- PING is a connection-level frame (RFC 9113 sec 6.7). Handler is installed on the TCP parent channel, not H2 stream channels. Detected via Http2MultiplexHandler presence in pipeline.

2. Idempotent installation -- in reactor-netty 1.2.x doOnConnected fires on the parent's State.CONFIGURED event only (not per-stream); child H2 stream channels emit STREAM_CONFIGURED separately. The handler still resolves to the parent channel and uses a pipeline-name-based installIfAbsent guard (consistent with Http2ParentChannelExceptionHandler) as defense-in-depth in case the contract changes. Concurrent races are caught via IllegalArgumentException (benign).

3. Idle-based gating (no active-stream check) -- PING is sent only when (now - lastActivityNanos) >= pingInterval. lastActivityNanos is refreshed on every inbound frame the parent pipeline sees in channelRead -- which is PING ACKs, SETTINGS, and GOAWAY only (HTTP request/response frames are dispatched by Http2MultiplexHandler to child stream channels and do not surface here). It is also refreshed when a PING is sent, so consecutive PINGs naturally pace at pingInterval apart. This is why no numActiveStreams() inspection is needed: PING cadence is anchored to PING activity itself, not to request traffic.

4. At-most-one outstanding PING -- pingOutstandingSinceNanos gates to 1 in-flight. On timeout, resets to 0 to allow next retry.

5. Consecutive failure threshold -- default 5. Tolerates transient blips; on each timeout or write failure: increment counter, unblock next PING, close only at threshold. Each round is ≈ pingTimeout + tick-detection lag ≈ 2.5s, so worst case is ≈ 12–15s with defaults. Tune lower via COSMOS.HTTP2_PING_FAILURE_THRESHOLD for tighter detection. Counting write failures (not just ACK timeouts) closes a state-machine gap where a channel stuck with persistently failing writes but channel.isActive()==true (e.g. H2 codec rejecting frames, stalled flow-control) would otherwise loop forever without ever closing.

6. No sharding needed -- Rust SDK shards because hyper opens 1 connection per client. reactor-netty natively pools multiple H2 connections via Http2AllocationStrategy.

7. Runtime toggle -- Configs.isHttp2PingHealthEnabled() is re-checked on every periodic tick. When flipped to false the tick becomes a no-op and clears any in-flight PING state, but the scheduled timer is kept alive so flipping it back to true resumes PINGing on the same connection without needing a reconnect.

8. H2-only registration -- doOnConnected PING handler registration is inside if (isH2Enabled) branch to avoid per-connect overhead for HTTP/1.1 clients.

Configuration

Each setting is resolved with the standard System.getProperty -> System.getenv -> default chain (matching the existing HTTP2_ENABLED family), so both forms below work.

System Property	Env Variable	Default	Description
COSMOS.HTTP2_PING_HEALTH_ENABLED	COSMOS_HTTP2_PING_HEALTH_ENABLED	true	Master switch
COSMOS.HTTP2_PING_INTERVAL_IN_SECONDS	COSMOS_HTTP2_PING_INTERVAL_IN_SECONDS	1	Idle threshold before PING
COSMOS.HTTP2_PING_TIMEOUT_IN_SECONDS	COSMOS_HTTP2_PING_TIMEOUT_IN_SECONDS	2	ACK timeout per attempt
COSMOS.HTTP2_PING_FAILURE_THRESHOLD	COSMOS_HTTP2_PING_FAILURE_THRESHOLD	5	Consecutive timeouts before close (Java-specific; peer SDKs close on first timeout)

Changes

Production (azure-cosmos):

• Http2PingHandler -- ChannelDuplexHandler: idle tracking, PING send, ACK timeout, consecutive failure threshold (counting both ACK timeouts and write failures), connection close
• Configs -- 4 PING configuration getters using the System.getProperty -> System.getenv -> default resolution chain (parity with HTTP2_ENABLED-family settings)
• ReactorNettyClient -- installs handler via doOnConnected on H2 parent channels (inside if (isH2Enabled) branch)

Test (azure-cosmos-tests):

• Http2PingKeepaliveTest.connectionClosedOnPingTimeout -- network fault test (manual-http-network-fault group). Runs in both the manual-http-network-fault (thin-client, port 10250) and manual-http-network-fault-compute (Compute Gateway, port 443) Maven profiles; sudo detection lets it run as root in Docker and via sudo in CI.
• Http2PingHandlerTest -- unit tests for kill-switch state-clear (clears pingOutstandingSinceNanos and consecutiveFailures when flipped off), channelInactive cleanup (cancels the scheduled task), and the write-failure path (increments consecutiveFailures and closes at threshold when writeAndFlush fails).

CI:

• Cosmos_live_test_http2_network_fault stage in sdk/cosmos/tests.yml (Ubuntu-only, MaxParallel=1). PreSteps installs iptables for network fault injection.
• eng/pipelines/templates/stages/cosmos/live-http2-network-fault-platform-matrix.json -- two matrix entries (thin-client and Compute Gateway).
• HTTP/2 PING -D flags (HTTP2_ENABLED, HTTP2_PING_HEALTH_ENABLED, HTTP2_PING_INTERVAL_IN_SECONDS, HTTP2_PING_TIMEOUT_IN_SECONDS) are wired per-profile via <systemPropertyVariables> in azure-cosmos-tests/pom.xml rather than via stage-level AdditionalArgs. This prevents a CLI -D from overriding each profile's COSMOS.THINCLIENT_ENABLED value (SUREFIRE-121), which is what keeps the thin-client leg (THINCLIENT=true) and the Compute leg (THINCLIENT=false) actually distinct.

Testing: connectionClosedOnPingTimeout

What it proves: PINGs are actively flowing on idle connections, ACK timeouts are detected, and broken connections are closed.

Without PING probing, iptables -j DROP would go completely undetected -- channel.isActive() stays true, no GOAWAY arrives, no TCP RST is sent. The test would time out. The fact that the handler detects and closes the connection is definitive proof that PINGs are being sent, ACKs are being tracked, and the consecutive failure threshold works.

How it works:

1. Creates an H2 client with PING interval=1s, timeout=2s, failure threshold=2 (overridden from default 5 for faster test), and a single-connection pool (Http2ConnectionConfig API)
2. Does a warm-up read and records the initial parent channel ID
3. Installs an iptables -j DROP rule to silently discard all outbound TCP traffic to the Cosmos DB gateway port
4. Waits 10 seconds for the handler to detect the failure:
   • Round 1: PING sent, 2s timeout elapses, no ACK -- consecutiveFailures=1, handler retries
   • Round 2: PING sent, 2s timeout elapses, no ACK -- consecutiveFailures=2 >= threshold -- handler closes connection
5. Removes the iptables rule to restore connectivity
6. Does a recovery read which succeeds on a newly created connection
7. Asserts: the recovery channel ID is different from the initial one

Local Docker test evidence

Http2PingKeepaliveTest#connectionClosedOnPingTimeout was run end-to-end in Docker (--cap-add=NET_ADMIN) against thin-client-mr-bs-ci for both transports the test now covers. Per-test config: interval=1s, timeout=2s, failureThreshold=2, single-conn pool.

Thin-client / Gateway V2 variant (-Pmanual-http-network-fault, port 10250)

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.61 s
RESULT: thinClient=true, port=10250, initial=378fbef3, recovery=b52281bd, DIFFERENT_CONNECTION=true

Key log lines for the affected channel 378fbef3:

22:09:57,552 DEBUG Http2PingHandler installed on channel [id: 0x378fbef3, L:/172.17.0.2:38360 - R:thin-client-mr-bs-ci-westcentralus.documents.azure.com/13.71.194.4:10250], interval=1s, timeout=2s, checkEvery=500ms
22:09:58,553 DEBUG PING #1 sent on channel [id: 0x378fbef3, L:/172.17.0.2:38360 - R:.../13.71.194.4:10250]
22:10:01,052 DEBUG PING ACK timeout on channel [id: 0x378fbef3, ... R:.../13.71.194.4:10250] (attempt 1/2) -- will retry
22:10:01,552 DEBUG PING #2 sent on channel [id: 0x378fbef3, ... R:.../13.71.194.4:10250]
22:10:03,552 INFO  PING ACK not received for 2 consecutive attempts on channel [id: 0x378fbef3, ... R:.../13.71.194.4:10250] -- closing connection
22:10:09,327 DEBUG Http2PingHandler installed on channel [id: 0xb52281bd, ... R:.../13.71.194.4:10250], interval=1s, timeout=2s, checkEvery=500ms   (recovery)

Compute Gateway variant (-Pmanual-http-network-fault-compute, port 443)

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.93 s
RESULT: thinClient=false, port=443, initial=b1d3461e, recovery=b3c9191a, DIFFERENT_CONNECTION=true

Key log lines for the affected channel b1d3461e (IP-scoped DROP to the resolved regional GW IP):

22:15:52,031 DEBUG Http2PingHandler installed on channel [id: 0xb1d3461e, L:/172.17.0.2:50116 - R:thin-client-mr-bs-ci-westcentralus.documents.azure.com/13.71.194.4:443], interval=1s, timeout=2s, checkEvery=500ms
22:15:53,034 DEBUG PING #1 sent on channel [id: 0xb1d3461e, ... R:.../13.71.194.4:443]
22:15:55,532 DEBUG PING ACK timeout on channel [id: 0xb1d3461e, ... R:.../13.71.194.4:443] (attempt 1/2) -- will retry
22:15:56,033 DEBUG PING #2 sent on channel [id: 0xb1d3461e, ... R:.../13.71.194.4:443]
22:15:58,032 INFO  PING ACK not received for 2 consecutive attempts on channel [id: 0xb1d3461e, ... R:.../13.71.194.4:443] -- closing connection
22:16:04,259 DEBUG Http2PingHandler installed on channel [id: 0xb3c9191a, ... R:.../13.71.194.4:443], interval=1s, timeout=2s, checkEvery=500ms   (recovery)

Signal summary

Signal	Log line	Both variants
Handler install	Http2PingHandler installed on channel [id: 0x..., L:..., R:...], interval=1s, timeout=2s, checkEvery=500ms (DEBUG)	yes
PING sent	PING #N sent on channel [id: 0x..., L:..., R:...] (DEBUG)	yes
PONG/ACK received	intentionally not logged — silent reset of outstanding counter; proven by unrelated channels' monotonically advancing PING #N over the same window	yes
Channel closed on timeout	PING ACK not received for 2 consecutive attempts on channel [id: 0x..., ...] -- closing connection (INFO)	yes
Recovery on new channel	Test assertion initial != recovery (DIFFERENT_CONNECTION=true)	yes

Channel identifier format (full Netty Channel.toString() including L: local and R: remote address) matches Http2ParentChannelExceptionHandler so PING events can be correlated with parent-channel exceptions and pool-eviction logs.

[Copilot]

Pull request overview

Adds a custom HTTP/2 PING keepalive handler installed on H2 parent channels to prevent middlebox idle-connection reaping and to detect broken connections. The handler sends PINGs at a configurable interval while idle and closes the channel after a configurable number of consecutive PING ACK timeouts. New system-property knobs are added to Configs, and a new CI stage (Cosmos_Live_Test_Http2NetworkFault) runs iptables-based fault-injection tests.

Changes:

• New Http2PingHandler (idle tracking, PING send, ACK timeout, consecutive-failure close) and wiring via ReactorNettyClient.doOnConnected.
• Four new Configs properties: HTTP2_PING_HEALTH_ENABLED, HTTP2_PING_INTERVAL_IN_SECONDS, HTTP2_PING_TIMEOUT_IN_SECONDS, HTTP2_PING_FAILURE_THRESHOLD.
• New Http2PingKeepaliveTest (manual-http-network-fault group), Maven profile, TestNG suite, matrix JSON, and tests.yml stage with iproute2/iptables install.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/Http2PingHandler.java	New handler implementing idle-based PING, ACK timeout tracking, consecutive-failure close.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ReactorNettyClient.java	Adds doOnConnected that installs the PING handler on H2 parent channels; minor refactor to accessor variable.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java	Adds 4 system properties + getters for PING enable/interval/timeout/failure-threshold.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/Http2PingKeepaliveTest.java	Two tests: idle PING keepalive and iptables-induced PING timeout closing the connection.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java	Adds manual-http-network-fault group to before/after suite annotations.
sdk/cosmos/azure-cosmos-tests/src/test/resources/manual-http-network-fault-testng.xml	TestNG suite for the new group.
sdk/cosmos/azure-cosmos-tests/pom.xml	New manual-http-network-fault Maven profile bound to the new TestNG suite.
sdk/cosmos/tests.yml	New Cosmos_Live_Test_Http2NetworkFault pipeline stage with iptables/tc pre-steps.
sdk/cosmos/live-http2-network-fault-platform-matrix.json	Matrix config for the new pipeline stage.

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

@sdkReviewAgent

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

✅ Review complete (45:33)

Posted 5 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[FabianMeiswinkel]

LGTM - Thanks!

[jeet1995]

Benchmark Results: HTTP/2 Ping Keepalive (Corrected)

Branch: squad/http2-ping-keepalive (jeet1995/azure-sdk-for-java @ 24c8203b8fd)
Baseline: main (Azure/azure-sdk-for-java @ 3ed5b63f2)
VM: Standard_D2s_v6 (2 vCPU, 8 GB), West US
Cosmos Account: thin-client-failover-test (Gateway mode, Eventual consistency)
Config: 100k ops, 10 min max per scenario, HTTP/2 enabled
TC = Thin Client (COSMOS.THINCLIENT_ENABLED=true)

ReadThroughput (Document/Read)

Configuration	c20 ops/s	c50 ops/s	c100 ops/s	c20 lat (ms)	c50 lat (ms)	c100 lat (ms)	RU/op
Feature (HTTP/2)	813.8	2029.9	4049.8	24.53	24.42	24.67	1.0
Feature (HTTP/2+TC)	825.4	2065.0	4040.4	24.05	24.16	24.63	2.0
Main (HTTP/2)	820.1	2030.5	4010.9	24.33	24.42	25.0	1.0
Main (HTTP/2+TC)	826.9	2065.8	3998.6	24.05	24.15	24.89	2.0

WriteThroughput (Document/Create)

Configuration	c20 ops/s	c50 ops/s	c100 ops/s	c20 lat (ms)	c50 lat (ms)	c100 lat (ms)	RU/op
Feature (HTTP/2)	709.8	1777.4	3505.1	28.13	28.08	28.52	7.81
Feature (HTTP/2+TC)	720.6	1802.6	3527.1	27.71	27.69	28.45	7.81
Main (HTTP/2)	713.2	1775.7	3514.0	28.0	28.11	28.44	7.81
Main (HTTP/2+TC)	716.6	1781.8	3528.6	27.87	27.78	28.36	7.81

QuerySingle (Document/Query)

Configuration	c20 ops/s	c50 ops/s	c100 ops/s	c20 lat (ms)	c50 lat (ms)	c100 lat (ms)	RU/op
Feature (HTTP/2)	792.4	1962.5	3746.1	25.18	25.43	28.31	2.82
Feature (HTTP/2+TC)	805.3	1984.1	3791.1	24.63	25.12	28.52	5.65
Main (HTTP/2)	793.8	1960.3	3763.6	25.06	25.45	28.2	2.82
Main (HTTP/2+TC)	802.9	1995.0	3802.7	24.7	25.0	28.09	5.65

Delta Analysis (Feature vs Main)

Scenario	Concurrency	HTTP/2 Δ tput	HTTP/2+TC Δ tput	HTTP/2 Δ lat	HTTP/2+TC Δ lat
ReadThroughput	20	-0.8%	-0.2%	+0.8%	0.0%
ReadThroughput	50	-0.0%	-0.0%	0.0%	+0.0%
ReadThroughput	100	+1.0%	+1.0%	-1.3%	-1.0%
WriteThroughput	20	-0.5%	+0.6%	+0.5%	-0.6%
WriteThroughput	50	+0.1%	+1.2%	-0.1%	-0.3%
WriteThroughput	100	-0.3%	-0.0%	+0.3%	+0.3%
QuerySingle	20	-0.2%	+0.3%	+0.5%	-0.3%
QuerySingle	50	+0.1%	-0.5%	-0.1%	+0.5%
QuerySingle	100	-0.5%	-0.3%	+0.4%	+1.5%

Summary

✅ All deltas within ±1.5% — no measurable performance impact from HTTP/2 ping keepalive.

• Throughput and latency are effectively identical between feature and baseline across all 9 scenarios × 2 modes (18 comparisons).
• Zero errors (no GOAWAYs, stream resets, 429s, or retries) in any run.
• Thin client mode doubles RU for reads (1→2) and queries (2.82→5.65) vs standard — this is expected gateway-side behavior, not related to this PR.

Positive throughput delta = feature is faster. Negative latency delta = feature is faster.
Thin client runs confirmed via useThinClient: true in SDK logs.

[jeet1995]

/azp run java - cosmos - spark

[jeet1995]

/azp run java - cosmos - kafka

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[kushagraThapar]

Re-approving on 5b70c7c. Pulled the delta from my prior approval at c7dc195 (11 commits) and the PR is in noticeably better shape — flagging what landed for posterity:

Closures from my prior review:

• ✅ M1 / M2 — 5aaf3e8e adds killSwitchOff_clearsOutstandingPingState and channelInactive_cancelsPingTask unit tests. Both target exactly the invariants I called out (stale-state clear on kill-switch flip, scheduled task cancellation on channel close).
• ✅ H4 — 1835b558 moves the HTTP/2 PING + thin-client flags out of the stage-level AdditionalArgs in sdk/cosmos/tests.yml and into per-profile <systemPropertyVariables> in azure-cosmos-tests/pom.xml. The Compute matrix leg now routes to the regional gateway on :443 as intended — Maven -D precedence no longer forces every leg into thin-client.
• ✅ H2 (partial) — c6120154 rewrites the Configs.java header comment so the "single missed PING round ≈ 3s, close after threshold ≈ 15s" narrative is internally consistent with the DEFAULT_HTTP2_PING_FAILURE_THRESHOLD = 5 doc 10 lines below. The misleading "detected within 3s" line is gone.

New fixes during this round (caught by @xinlian12, not in my prior list):

• 🟢 Write-failure now counts toward close threshold (b74b11697, Http2PingHandler.java:178-195). Previously a write failure only cleared pingOutstandingSinceNanos without incrementing consecutiveFailures — so a channel stuck in a state where writes always fail but isActive() stays true (H2 codec rejecting frames, stalled flow-control, queued ClosedChannelException not propagated) would loop forever. Good catch; matching unit test writeFailure_incrementsConsecutiveFailuresAndClosesAtThreshold covers it.
• 🟢 TOCTOU guard on customHeaderCleaner (c6120154, ReactorNettyClient.java:194-205). The new get("customHeaderCleaner") == null check could race with concurrent doOnConnected callbacks; now wrapped in try { ... } catch (IllegalArgumentException ignored) matching the sibling exHandler pattern. Safe and minimal.
• 🟢 ACK-timeout cascade test (c6120154, Http2PingHandlerTest:ackTimeout_incrementsConsecutiveFailuresAndClosesAtThreshold). The primary operational failure path (PING sent → no ACK within pingTimeoutNanos → after N timeouts close the channel) now has explicit coverage. Time is sidestepped via pingOutstandingSinceNanos reflection — clean approach.

Still-open suggestions (non-blocking, leaving these as you like):

• H1 — PR body §3 still claims "active-stream suppression via numActiveStreams() > 0". grep still finds zero refs in Http2PingHandler. Either strike the claim or add the suppression — your call.
• H3 — ctx.close() in the failure path is still a region-failover signal once consumed downstream. Wrapping in Http2PingTimeoutChannelClosedException would let WebExceptionUtility distinguish "we tore down the connection on purpose" from "remote slammed the door"; nice-to-have for diagnostics.
• H5 — The customHeaderCleaner idempotency get(...) == null guard (the pre-existing duplicate-add fix you added in this PR) still lacks an explicit one-line comment in the PR body / changelog. Now even more worth calling out since c6120154 added the matching TOCTOU catch.

Verified-negatives (still hold at HEAD):

• @Sharable is intentional and safe for this stateless installer pattern.
• handlerRemoved / channelInactive cleanup is correct and now unit-tested.
• Late-ACK guards still in place.
• No public API surface change (env vars + a UserAgentFeatureFlags bit, both internal).

CI: The two red legs (Spark Databricks integration and StrongSession_Strong_Tcp_Fast) are TCP / Spark legs — unrelated to HTTP/2 PING which is HTTP-only. Worth retrying or letting the on-duty triage them, but not a blocker.

Approving — ship it whenever 🚀

[jeet1995]

@sdkReviewAgent

[xinlian12]

LGTM, great PR, thanks

[xinlian12]

✅ Review complete (50:38)

Posted 4 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[FabianMeiswinkel]

LGTM