Proactive connection management for broken connections#4
Merged
jeet1995 merged 1 commit intoApr 19, 2023
Conversation
jeet1995
merged commit cc9d003
into
jeet1995:ProactiveConnectionManagementForBrokenConnections
jeet1995
added a commit
that referenced
this pull request
Apr 9, 2026
…tom PING - Http2ConnectTimeoutBifurcationTests: use NetworkFaultInjector for sudo detection, iptables helpers, and cleanup. Remove duplicated methods. Per-port delay methods (addPerPortDelay, addPerPortSynDelay) kept locally as they are bifurcation-test-specific. - SPEC: Fix Design Choices #3 and #4 to reflect custom Http2PingHandler (not native pingAckTimeout). Fix Architecture diagram, Config table, Non-Goals, and .NET Parity sections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeet1995
added a commit
that referenced
this pull request
Apr 10, 2026
…unts In multi-writer accounts, refreshLocationPrivateAsync() stops the background refresh timer when shouldRefreshEndpoints() returns false. This means topology changes (e.g., multi-write to single-write transitions) go undetected until the next explicit refresh trigger. The .NET SDK (azure-cosmos-dotnet-v3) correctly continues the background refresh loop unconditionally - the loop only stops when canRefreshInBackground is explicitly false, not when shouldRefreshEndpoints returns false. This fix adds startRefreshLocationTimerAsync() to the else-branch of refreshLocationPrivateAsync(), ensuring the background timer always reschedules itself regardless of whether endpoints currently need refreshing. Without this fix, after a multi-write -> single-write -> multi-write transition, reads remain stuck on the primary region because the SDK never re-reads account metadata to learn about the restored multi-write topology. Related: PR Azure#6139 (point #4 in description acknowledged this bug) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeet1995
added a commit
that referenced
this pull request
Apr 10, 2026
…unts In multi-writer accounts, refreshLocationPrivateAsync() stops the background refresh timer when shouldRefreshEndpoints() returns false. This means topology changes (e.g., multi-write to single-write transitions) go undetected until the next explicit refresh trigger. The .NET SDK (azure-cosmos-dotnet-v3) correctly continues the background refresh loop unconditionally - the loop only stops when canRefreshInBackground is explicitly false, not when shouldRefreshEndpoints returns false. This fix adds startRefreshLocationTimerAsync() to the else-branch of refreshLocationPrivateAsync(), ensuring the background timer always reschedules itself regardless of whether endpoints currently need refreshing. Without this fix, after a multi-write -> single-write -> multi-write transition, reads remain stuck on the primary region because the SDK never re-reads account metadata to learn about the restored multi-write topology. Unit tests updated: - backgroundRefreshForMultiMaster: assertTrue (timer must keep running) - backgroundRefreshDetectsTopologyChangeForMultiMaster: new test proving MW->SW transition detection via mock Related: PR Azure#6139 (point #4 in description acknowledged this bug) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeet1995
added a commit
that referenced
this pull request
Apr 15, 2026
…unts In multi-writer accounts, refreshLocationPrivateAsync() stops the background refresh timer when shouldRefreshEndpoints() returns false. This means topology changes (e.g., multi-write to single-write transitions) go undetected until the next explicit refresh trigger. The .NET SDK (azure-cosmos-dotnet-v3) correctly continues the background refresh loop unconditionally - the loop only stops when canRefreshInBackground is explicitly false, not when shouldRefreshEndpoints returns false. This fix adds startRefreshLocationTimerAsync() to the else-branch of refreshLocationPrivateAsync(), ensuring the background timer always reschedules itself regardless of whether endpoints currently need refreshing. Without this fix, after a multi-write -> single-write -> multi-write transition, reads remain stuck on the primary region because the SDK never re-reads account metadata to learn about the restored multi-write topology. Unit tests updated: - backgroundRefreshForMultiMaster: assertTrue (timer must keep running) - backgroundRefreshDetectsTopologyChangeForMultiMaster: new test proving MW->SW transition detection via mock Related: PR Azure#6139 (point #4 in description acknowledged this bug) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeet1995
added a commit
that referenced
this pull request
Jun 10, 2026
…in lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeet1995
added a commit
that referenced
this pull request
Jul 9, 2026
…V2 endpoint (Azure#47759) * Route Execute Stored Procedure requests to Thin Proxy endpoint. * Route Execute Stored Procedure and QueryPlan requests to Thin Proxy endpoint. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Adding query + thin-client tests. * Fixing tests. * Fixing tests. * Fixing tests. * Addressing review comments. * Addressing review comments. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add change feed tests for FeedRange.forFullRange and forLogicalPartition Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add thin client E2E test matrix documentation for QueryPlan review Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix testGetCurrentDateTime flaky assertion, add AAD auth support, RNTBD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor thin client query tests for reliability - Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix container leaks and Direct TCP AAD auth in tests - Fix container leak: get container reference before createContainer() so finally block can always delete. Use safeDeleteContainer() helper. - Fix Direct TCP client: apply AAD credential via applyCredential() for accounts with disableLocalAuth=true. All 89 thin client tests pass (80 query + 3 change feed + 3 point ops + 3 sprocs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use bulkDelete in AfterClass for seeded docs cleanup Replace sequential deleteItem loop with executeBulkOperations for faster and more reliable cleanup of seeded test documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback 1. Remove rntbd.instructions.md (not part of PR) 2. Move MAPPER variable to top of PartitionKeyInternalTest 3. Remove COSMOS.REUSE_DATABASE_ID (beforeSuiteReuse, afterSuite guard, pom.xml) 4. Keep System.setProperty in ThinClientQueryE2ETest (required — extends TestSuiteBase not ThinClientTestBase, property must be set before client build) 5. Fix RNTBD IDs to match server-side RntbdConstants.cs (ADO PR 1982503): SupportedQueryFeatures=0x00FF (String), QueryVersion=0x0100 (SmallString) The SmallString type was the root cause of 400 BadRequest — String uses 2-byte length prefix, SmallString uses 1-byte, causing token stream misparse. All 89 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align vector/FTS/hybrid tests with existing patterns - Add euclidean VectorDistance variant to vector search test - Add document ID comparison to FTS and hybrid tests (was count-only) - Add testFullTextScoreRanking: ORDER BY RANK FullTextScore with exact order comparison between Direct and thin client - All assertions now validate both result size AND content parity 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments + fix broken link Code fixes: - PartitionKeyInternalHelper: materialize fieldNames() Iterator in error msg - QueryPlanRetriever: throw IllegalStateException when partitionKeyDef null in thin client mode (was silent fallthrough) - RxDocumentClientImpl: add parentheses around compound && in boolean exprs - ThinClientStoreModel: use != instead of !(==) - ThinClientTestBase: remove stale 'uncomment' comment - ThinClientQueryE2ETest: accept status 0 (transport rejection) for invalid query — proxy returns transport error, not HTTP 400 - THINCLIENT_TEST_MATRIX.md: remove broken link, update methodology to reflect Direct TCP baseline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments (round 2) - RntbdOperationType: add QueryPlan case to fromId()/fromType() lookups - PartitionKeyInternalHelper: sanitize PII from error message (log node type instead of raw partition key values) - THINCLIENT_TEST_MATRIX.md: remove PR-specific references and known blockers - PartitionedQueryExecutionInfo: remove stale queryRanges from backing ObjectNode after EPK conversion to prevent data inconsistency - QueryPlanRetriever: add TODO for 3 missing query features vs .NET SDK (ListAndSetAggregate, CountIf, HybridSearchSkipOrderByRewrite) 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor queryRanges deserialization — detect format from response, no ObjectNode mutation - PartitionedQueryExecutionInfo: getQueryRanges() now inspects the first range's 'min' field to detect format (string=EPK hex, array=PK-internal) and applies the correct deserialization path automatically. - Remove ObjectNode.remove('queryRanges') — no longer mutate the backing JSON. The format is detected lazily on first access. - 3-arg constructor now takes PartitionKeyDefinition (not pre-computed ranges) so conversion happens inside getQueryRanges(), keeping the DTO self-contained. - QueryPlanRetriever: pass partitionKeyDefinition to constructor instead of pre-converting ranges externally. 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use agnostic QueryRangesFormat hint instead of PartitionKeyDefinition presence Introduce QueryRangesFormat enum (EPK_HEX_STRING, PARTITION_KEY_INTERNAL_ARRAY) as the deserialization hint. The hint guides which path to try first, but getQueryRanges() always validates by inspecting the actual min node type — if the hint doesn't match the wire format, it falls through to the other path. Naming is transport-agnostic (no Proxy/ThinClient/Gateway references). PartitionKeyDefinition is only stored when needed for EPK conversion, not used as a format signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fetchQueryPlanForValidation caller after merge Pass null for the DocumentCollection parameter so the public fetchQueryPlanForValidation entry point compiles against the new fetchQueryPlan signature introduced during the merge with upstream/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PartitionedQueryExecutionInfo(ObjectNode, RequestTimeline) ctor public The upstream ReadManyByPartitionKeyQueryPlanValidationTest (added in Azure#48801) constructs PartitionedQueryExecutionInfo directly from a test package. Upstream/main exposed this ctor as public; restore that visibility so the new test compiles after the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Cosmos] Default thin-client to enabled and add HTTP/2 connectivity-probe gate Adds an EndpointOrchestrator that fans out POST /connectivity-probe to every thin-client regional endpoint after each topology refresh. SDK only routes data-plane traffic to thin-client (Gateway V2) when all regional probes succeed across N consecutive refresh cycles (configurable via COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD, default 2); otherwise traffic falls back to Gateway V1 at the next refresh boundary. No mid-flight fallback. Caveats: - Probe wiring is skipped entirely for Direct mode and when HTTP/2 is not configured; controlled by RxDocumentClientImpl.useThinClient. - QueryPlan, metadata reads, and AllVersionsAndDeletes change feed continue to route through Compute Gateway (Gateway V1). - Probe failures are absorbed inside the orchestrator and the trigger is fire-and-forget on the GEM scheduler, so probe issues can never trip CosmosClient initialization or fail a topology refresh. - EndpointOrchestrator implements Closeable and is closed from GlobalEndpointManager.close() so no further probes are issued after client shutdown. THINCLIENT_ENABLED now defaults to true; opt out via COSMOS.THINCLIENT_ENABLED=false or COSMOS.THINCLIENT_PROBE_ENABLED=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 review comments: probe single-flight, body-drain lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 second-batch review feedback - LocationCache.getThinClientRegionalEndpoints now walks both read and write region endpoint maps so single-master write-region failures still flip the probe gate. - EndpointOrchestrator.forceUnhealthy(reason) provides a non-HTTP path to flip the gate; GlobalEndpointManager calls it when topology says thin-client is eligible but no regional endpoint resolves. - Symmetric hysteresis: new COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD (default 1) so operators can require N consecutive GREEN cycles before flipping back to proxy. - Extracted RxDocumentClientImpl.useThinClientStoreModel(...) body into package-private static shouldUseThinClientStoreModel for direct unit testability; added ThinClientRoutingGateTests covering 9 routing paths. - EndpointOrchestratorTests.stubResponse now returns Mono.empty() to avoid Unpooled.EMPTY_BUFFER refCnt underflow across multiple probe calls. - Removed unused locals; added recoveryThresholdRequiresMultipleGreenCycles, forceUnhealthy_flipsGateToRedWithoutRunningProbe, forceUnhealthy_onClosedOrchestrator_isNoOp tests. All 57 unit tests in the touched files pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify CHANGELOG: probe recovery threshold is now configurable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 review (rounds 4-8): reactor-chain probe, FQN cleanup, fix gwV2Cto and ThinClient user-agent assertions - GlobalEndpointManager: convert thin-client probe trigger to a Mono<Void> chained into the topology-refresh reactor pipeline (replaces fire-and-forget subscribe). Removes thinClientProbeDisposable field and its close() handling since cancellation now propagates through the outer subscription. - EndpointProbeClient/EndpointProbeClientTests/ThinClientProbeWiringTests: replace inline FQNs with imports (java.io.Closeable, java.util.List, java.net.ConnectException, com.azure.cosmos.implementation.http.HttpHeaders). - ClientConfigDiagnosticsTest: compute gwV2Cto dynamically from Configs.isThinClientEnabled() so assertions remain valid after the default flip to true. - ConfigsTests: update default-threshold assertions from 2 to 1 to match DEFAULT_THINCLIENT_PROBE_FAILURE_THRESHOLD=1. - UserAgentContainerTest.UserAgentIntegration: expect '|F4' suffix because the ThinClient feature flag (1 << 2) is now included by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test fallout from default ThinClient enablement UserAgentSuffixTest.validateUserAgentSuffix and CosmosDiagnosticsTest.generateHttp2OptedInUserAgentIfRequired: include UserAgentFeatureFlags.ThinClient in computed |F<hex> suffix when COSMOS.THINCLIENT_ENABLED is true (now default after Gateway V2 default enablement). Mirrors RxDocumentClientImpl.addUserAgentSuffix + UserAgentContainer.setFeatureEnabledFlagsAsSuffix behavior. SinglePartitionDocumentQueryTest.querySinglePartitionDocuments: spy on both gateway-proxy and thin-proxy and assert exactly one invocation. Previous code only spied on the proxy implied by useThinClient() config intent, which races with the probe-healthy gate -- routing AND's intent with isProxyProbeHealthy() so on first cycle the request may go through gateway even when thin-client is configured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe in Http2PingKeepaliveTest The test installs an iptables DROP on thin-client port 10250 to verify that Http2PingHandler closes the broken connection after consecutive PING ACK timeouts and the recovery request uses a new connection on the same regional endpoint. After default Gateway V2 enablement, the connectivity probe also fires HTTP/2 POSTs to port 10250 on every account refresh. With iptables dropping that port, the probe trips proxyHealthy=false, useThinClient StoreModel() returns false, and the data plane request routes through Gateway V1 on port 443 -- which iptables is not dropping. Result: the PING handler never fires, the warm-up and recovery requests use the same gateway channel, and the assertion 'recovery channel must differ from initial' fails (both ended up as 77af2e47 on build 6419227). Set COSMOS.THINCLIENT_PROBE_ENABLED=false in beforeClass so the probe short-circuits to a no-op, EndpointProbeClient.proxyHealthy stays optimistically true, and the data plane request actually flows over port 10250 where the iptables DROP can take effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe by default for E2E tests in TestSuiteBase Build 6424287 surfaced two new failure patterns: 1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures) - asserts substatus 1003 from the thin-client routing path, but observed 0 because the data plane was routed to Gateway V1 instead of the proxy. 2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures) - TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained ':10250/', i.e. nothing actually went to the thin-client proxy. Both patterns trace to the same source: the new connectivity probe is enabled by default, the proxy-side /connectivity-probe endpoint is not deployed in every CI test account yet, and the default failure threshold is 1. So after the first probe cycle the SDK marks the proxy unhealthy and routes data plane traffic to Gateway V1, which breaks tests that explicitly assert thin-client routing. Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic, configuration-driven routing. Tests that exercise the probe itself (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are not affected. Also drop the now-redundant per-class override in Http2PingKeepaliveTest - the base class disables it, and the test's @afterclass clear would otherwise re-enable the probe for any subsequent E2E test sharing the JVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Disable thin-client probe by default for E2E tests in TestSuiteBase" This reverts commit ad9e3df. * Add per-class thinclient probe disable in CosmosNotFoundTests and PerPartitionCircuitBreakerE2ETests Companion to the prior revert. The revert undid the global TestSuiteBase probe disable (which masked production behaviour). This commit adds the necessary per-class disable to the two test classes whose assertions explicitly require thinclient routing: CosmosNotFoundTests (thinclient group) and PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group). Both clear the property in their @afterclass. Http2PingKeepaliveTest already has its own disable (restored by the revert). Production callers continue to get the connectivity probe ON by default with the production failure threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add unit tests for QueryPlan and stored-procedure thin-client routing Covers 5 new scenarios in ThinClientRoutingGateTests: - ExecuteStoredProcedure on a StoredProcedure resource routes to thin client - Non-execute StoredProcedure ops (Create) route to Gateway V1 - OperationType.QueryPlan routes to thin client - QueryPlan returns false when probe is unhealthy - ExecuteStoredProcedure returns false when probe is unhealthy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove endpoint-probe content from QueryPlan PR branch Reverts the merge of jeet1995/thin-client-probe-flow that was brought into this PR earlier, while keeping the branch up-to-date with upstream/main. Net diff vs upstream/main is now only the QueryPlan and StoredProcedure Gateway V2 routing changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Thread DocumentCollection into readMany query-plan validation readManyByPartitionKeys validates any caller-supplied custom query by fetching a query plan and asserting it is single-partition and non-hybrid. Until now that validation called fetchQueryPlanForValidation with no DocumentCollection, so the plan request was pinned to Gateway V1 (useGatewayMode = (partitionKeyDefinition == null)). Thread the container's DocumentCollection from RxDocumentClientImpl.validateCustomQueryForReadManyByPartitionKeys -> DocumentQueryExecutionContextFactory.fetchQueryPlanForValidation so QueryPlanRetriever has the PartitionKeyDefinition needed to convert PartitionKeyInternal-formatted queryRanges from the thin client (Gateway V2) proxy into the EPK-hex Range<String> entries the query pipeline consumes. With this wiring, the validation query plan goes to the thin client when the client is configured for it, and remains on Gateway V1 otherwise. No behavior change on the non-thin-client path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client QueryPlan error returning statusCode 0 instead of 400 The thin-client/Gateway-V2 proxy returns a non-2xx response with a raw, non-JSON, NUL-padded error body for invalid-syntax queries. In RxGatewayStoreModel.validateOrThrow, new CosmosError(body) attempted to parse that body as JSON and threw IllegalArgumentException, which escaped the method before the existing status-carrying throw could run. Upstream then wrapped it as statusCode 0. Wrap the CosmosError(body) construction in a narrow try/catch (IllegalArgumentException) and fall back to the non-parsing CosmosError(errorCode, message) constructor with a sanitized body. The existing throw now fires with the correct status (400) and the proxy error text is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden ThinClientQueryE2ETest assertions (F1-F5) F1: strict 400 + unconditional thin-client endpoint check on invalid query, locking the statusCode-0->400 fix. F2: ordered-vs-sorted-set document-ID comparison (ORDER BY sequence-compared, others set-compared). F3: ID-set equality + no-duplicate check across drained continuation pages. F4: numeric-tolerance (1e-6) comparison for scalar and GROUP BY aggregates to avoid float-formatting false mismatches. F5: validated vector/full-text/hybrid queries match Direct vs thin-client end-to-end through the proxy. Validated live via -Pthinclient: 84 tests, 0 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace thin-client test matrix with reverse-engineered E2E test spec Rewrite THINCLIENT_TEST_MATRIX.md as a reviewable test-design specification reverse-engineered from the committed ThinClientQueryE2ETest code (84 tests). Documents the differential-testing oracle (Direct :443 baseline vs thin client :10250 SUT), the data-model fixture, every assertion contract (endpoint provenance, ordered-vs-unordered ID equality, scalar/GROUP BY tolerance), the full 84-test matrix, the F1-F5 hardened special cases (continuation draining, invalid-query 400, vector/FTS/hybrid ranking, readMany validation path), advertised-feature coverage, and known gaps (CountIf/DCount/MultipleOrderBy) for reviewer sign-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply Aditya Sarpotdar review feedback to thin-client E2E tests Executes actionable review feedback on ThinClientQueryE2ETest plus two harness fixes surfaced during live validation against thin-client-multi-region-ci. ThinClientQueryE2ETest: - Strict ordering: ORDER BY results validated with isStrictlyOrdered across the board (stricter-by-default, per reviewer guidance) instead of set parity. - Add testMultipleOrderBy() with a composite-index container to cover the MultipleOrderBy query feature. - Add testDCount() using the canonical Cosmos DCount idiom (COUNT over a DISTINCT VALUE subquery); SQL-standard COUNT(DISTINCT ...) is not valid Cosmos SQL grammar. - Reword multi-EPK-range comment/javadoc to reflect emulator/backend reality (multiple metadata ranges served by a single backend partition; SDK routing and query pipeline still exercised). Harness fixes: - TestSuiteBase: add wait-and-poll utility waitForCollectionToBeAvailableToRead (predicate on NotFound/substatus 1013) to deflake "Collection is not yet available for read" on freshly created containers; update call sites in OrderbyDocumentQueryTest, NonStreamingOrderByQueryVectorSearchTest, QueryValidationTests, ReadFeedCollectionsTest. - SinglePartitionDocumentQueryTest: make the processMessage Mockito assertion mode-aware (thin-client routes query plan + query => times(2)). Validated live against thin-client-multi-region-ci: 86/86 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: opt-in thin-client QueryPlan routing + kill-switch, stop advertising CountIf, remove PR-reference comments * Address review feedback: broaden gateway error fallback to ClassCastException, drop response ObjectNode mutation, fail-fast on empty queryRanges, align readMany routing test with opt-in * Make thin-client QueryPlan no-EPK-headers contract explicit in wrapInHttpRequest * Fix missing StandardCharsets import in RxGatewayStoreModelTest * Test: add QueryOracle-derived LIKE/scalar-expression thin-client parity tests; drain only first page in OperationPolicies readAllItems to avoid full-container enumeration timeout * Test: verify a cached proxy-generated query plan still executes correctly through the thin client * Test: cross-partition tests use a dedicated multi-physical-partition fixture and assert >1 partition key range contacted; enforce ordered full-row comparison when id is not projected; clearer naming * Add CHANGELOG entry for QueryPlan request routing to Gateway V2 * Call out Execute Stored Procedure support in CHANGELOG entry * Fix flaky region/timing-sensitive fault-injection E2E tests CustomerWorkflowPartitionLevelCircuitBreakerTest and CustomerWorkflowHighE2ETimeoutTest: bump the ThresholdBasedAvailabilityStrategy threshold from 100ms to 300ms so the cold primary-region connection can dispatch the request and record the injected fault before the availability-strategy hedge to a warm secondary region wins the race and cancels it (which previously yielded 0 fault hits). PerPartitionCircuitBreakerE2ETests: before asserting short-circuit routing, wait until the first preferred region is actually marked Unavailable, removing the race between the failure-counter estimate and the asynchronous routing exclusion. Scoped to single-faulty-region configs (expectedRegionCountWithFailures == 1) and run once so fault-in-all-regions configs (which can never short-circuit) do not hit the method timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove test AAD auth toggle; move thin-client test matrix to docs/ and align with ThinClientQueryE2ETest * Add cross-partition aggregate / GROUP BY parity tests for thin-client query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix EPK over-span; add QueryPlan parity tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enforce QueryPlan thin-client routing invariant and harden container-readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add kill-switch flip validation for QueryPlan thin-client routing Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore ambient COSMOS.THINCLIENT_ENABLED in ThinClientTestBase teardown ThinClientTestBase teardown unconditionally cleared the COSMOS.THINCLIENT_ENABLED system property. Since Configs.isThinClientEnabled() reads the property lazily per client build, a ThinClientTestBase subclass running before a property-dependent class inherited from main (e.g. CosmosMultiHashTest, ThinClientQueryE2ETest) in the same JVM wiped the ambient -DCOSMOS.THINCLIENT_ENABLED=true flag supplied by the thin-client CI lane. Later clients then built with thin client disabled and routed to :443, breaking the :10250 thin-client endpoint assertions. Save the prior value on enable and restore it on teardown (clear only when it was originally absent), mirroring CosmosConsistencyOverrideValidationTest's restoreSystemProperty pattern. Helpers changed from static to instance so the saved value is per-instance, avoiding races across TestNG factory instances. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove redundant COSMOS.THINCLIENT_ENABLED mutation from ThinClientTestBase; revert workflow tests to main ThinClientTestBase no longer sets/clears COSMOS.THINCLIENT_ENABLED per class. The 'thinclient' Maven lane already sets THINCLIENT_ENABLED=true and HTTP2_ENABLED=true JVM-wide via failsafe systemPropertyVariables, and all thin classes run only in that lane (groups={thinclient}). The prior per-class clearProperty in @afterclass wiped the ambient -D for later byte-identical-to-main classes in the same JVM, mis-routing requests to :443. Relying on the lane flag (matching ThinClientQueryE2ETest) removes that footgun. Also revert CustomerWorkflowHighE2ETimeoutTest and CustomerWorkflowPartitionLevelCircuitBreakerTest to upstream/main (availability-strategy threshold stays 100ms), since this PR requires no changes there. Verified locally (AZURE_TEST_MODE=LIVE, -Pthinclient,query,consistency-overrides): 234 passed, 0 failures/skips; QueryPlan routes to :10250. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Abhijeet Mohanty <copilot@noreply.local>
jeet1995
added a commit
that referenced
this pull request
Jul 13, 2026
…probe mechanism. (Azure#49437) * Route Execute Stored Procedure requests to Thin Proxy endpoint. * Route Execute Stored Procedure and QueryPlan requests to Thin Proxy endpoint. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Adding query + thin-client tests. * Fixing tests. * Fixing tests. * Fixing tests. * Addressing review comments. * Addressing review comments. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add change feed tests for FeedRange.forFullRange and forLogicalPartition Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add thin client E2E test matrix documentation for QueryPlan review Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix testGetCurrentDateTime flaky assertion, add AAD auth support, RNTBD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor thin client query tests for reliability - Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix container leaks and Direct TCP AAD auth in tests - Fix container leak: get container reference before createContainer() so finally block can always delete. Use safeDeleteContainer() helper. - Fix Direct TCP client: apply AAD credential via applyCredential() for accounts with disableLocalAuth=true. All 89 thin client tests pass (80 query + 3 change feed + 3 point ops + 3 sprocs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use bulkDelete in AfterClass for seeded docs cleanup Replace sequential deleteItem loop with executeBulkOperations for faster and more reliable cleanup of seeded test documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback 1. Remove rntbd.instructions.md (not part of PR) 2. Move MAPPER variable to top of PartitionKeyInternalTest 3. Remove COSMOS.REUSE_DATABASE_ID (beforeSuiteReuse, afterSuite guard, pom.xml) 4. Keep System.setProperty in ThinClientQueryE2ETest (required — extends TestSuiteBase not ThinClientTestBase, property must be set before client build) 5. Fix RNTBD IDs to match server-side RntbdConstants.cs (ADO PR 1982503): SupportedQueryFeatures=0x00FF (String), QueryVersion=0x0100 (SmallString) The SmallString type was the root cause of 400 BadRequest — String uses 2-byte length prefix, SmallString uses 1-byte, causing token stream misparse. All 89 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align vector/FTS/hybrid tests with existing patterns - Add euclidean VectorDistance variant to vector search test - Add document ID comparison to FTS and hybrid tests (was count-only) - Add testFullTextScoreRanking: ORDER BY RANK FullTextScore with exact order comparison between Direct and thin client - All assertions now validate both result size AND content parity 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments + fix broken link Code fixes: - PartitionKeyInternalHelper: materialize fieldNames() Iterator in error msg - QueryPlanRetriever: throw IllegalStateException when partitionKeyDef null in thin client mode (was silent fallthrough) - RxDocumentClientImpl: add parentheses around compound && in boolean exprs - ThinClientStoreModel: use != instead of !(==) - ThinClientTestBase: remove stale 'uncomment' comment - ThinClientQueryE2ETest: accept status 0 (transport rejection) for invalid query — proxy returns transport error, not HTTP 400 - THINCLIENT_TEST_MATRIX.md: remove broken link, update methodology to reflect Direct TCP baseline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments (round 2) - RntbdOperationType: add QueryPlan case to fromId()/fromType() lookups - PartitionKeyInternalHelper: sanitize PII from error message (log node type instead of raw partition key values) - THINCLIENT_TEST_MATRIX.md: remove PR-specific references and known blockers - PartitionedQueryExecutionInfo: remove stale queryRanges from backing ObjectNode after EPK conversion to prevent data inconsistency - QueryPlanRetriever: add TODO for 3 missing query features vs .NET SDK (ListAndSetAggregate, CountIf, HybridSearchSkipOrderByRewrite) 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor queryRanges deserialization — detect format from response, no ObjectNode mutation - PartitionedQueryExecutionInfo: getQueryRanges() now inspects the first range's 'min' field to detect format (string=EPK hex, array=PK-internal) and applies the correct deserialization path automatically. - Remove ObjectNode.remove('queryRanges') — no longer mutate the backing JSON. The format is detected lazily on first access. - 3-arg constructor now takes PartitionKeyDefinition (not pre-computed ranges) so conversion happens inside getQueryRanges(), keeping the DTO self-contained. - QueryPlanRetriever: pass partitionKeyDefinition to constructor instead of pre-converting ranges externally. 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use agnostic QueryRangesFormat hint instead of PartitionKeyDefinition presence Introduce QueryRangesFormat enum (EPK_HEX_STRING, PARTITION_KEY_INTERNAL_ARRAY) as the deserialization hint. The hint guides which path to try first, but getQueryRanges() always validates by inspecting the actual min node type — if the hint doesn't match the wire format, it falls through to the other path. Naming is transport-agnostic (no Proxy/ThinClient/Gateway references). PartitionKeyDefinition is only stored when needed for EPK conversion, not used as a format signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fetchQueryPlanForValidation caller after merge Pass null for the DocumentCollection parameter so the public fetchQueryPlanForValidation entry point compiles against the new fetchQueryPlan signature introduced during the merge with upstream/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PartitionedQueryExecutionInfo(ObjectNode, RequestTimeline) ctor public The upstream ReadManyByPartitionKeyQueryPlanValidationTest (added in #48801) constructs PartitionedQueryExecutionInfo directly from a test package. Upstream/main exposed this ctor as public; restore that visibility so the new test compiles after the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(cosmos): HTTP/2 PING handler must observe child-stream read activity, not just parent-pipeline frames Follow-up to #49095. Root cause: Http2PingHandler is installed on the parent (TCP) channel and only updated lastReadActivityNanos from parent-pipeline channelRead -- which sees PING ACK / SETTINGS / GOAWAY but NOT HEADERS / DATA frames. With reactor-netty's Http2MultiplexHandler in the parent pipeline, all application H2 traffic is demuxed onto per-stream child channels, so a connection serving thousands of QPS still looked idle to the PING handler, causing unnecessary PINGs and measurable P95 overhead under load. Fix: - New package-private AttributeKey<AtomicLong> LAST_CHILD_STREAM_READ_ACTIVITY_NANOS seeded on the parent channel in handlerAdded. - Http2PingCloseRewrapHandler (already installed in every child stream's pipeline) overrides channelReadComplete to stamp the parent attribute via accumulateAndGet(now, Math::max) -- once per read cycle, monotonic. - Http2PingHandler.maybeSendPing now uses effectiveLastReadActivityNanos = max(field, child attr) for both the idle-threshold check AND the outstanding-PING early-return: if child-stream reads arrived after an outstanding PING, clear pingOutstandingSinceNanos and reset consecutiveFailures (defense against middleboxes that drop PING-ACK but pass application H2 frames). - Names use lastReadActivity / read activity -- both signals are inbound reads, never writes. Tests: two new unit tests in Http2PingHandlerTest covering the suppression path and the outstanding-PING reset path; both bypass real time via reflection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * perf: install Http2PingCloseRewrapHandler in doOnConnected, not doOnRequest The previous .doOnRequest() chained onto the HTTP/2 HttpClient builder wrapped every outgoing request's Mono in a Reactor operator. That paid per-request costs even when the rewrap handler was already installed: * MonoPeekTerminal wrap + per-request subscriber allocation (InternalMonoOperator.subscribe +2.60% in alloc profile) * per-request BiConsumer lambda invocation (Http2Pool\$\...run +1.16% in alloc profile) * per-request channel pipeline String-key walk via pipeline.get(...) Long-run async-profiler diff vs v4.80.0 on D2 attributed the PR's -7.6% QPS / +14.5% mean / +166% tail-max regression primarily to a +10.2% allocation rate -- with the operators above dominating the delta. Move the install into the existing .doOnConnected() block, which already runs once per HTTP/2 child stream (the customHeaderCleaner install right above it relies on the same contract). The rewrap handler is @Sharable, so installing once per stream is correct and amortizes the install + pipeline-walk away from the hot path. The ch.parent() != null guard is kept so the install only targets child streams, never the parent TCP channel (which uses Http2ParentChannelExceptionHandler instead). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Cosmos] Default thin-client to enabled and add HTTP/2 connectivity-probe gate Adds an EndpointOrchestrator that fans out POST /connectivity-probe to every thin-client regional endpoint after each topology refresh. SDK only routes data-plane traffic to thin-client (Gateway V2) when all regional probes succeed across N consecutive refresh cycles (configurable via COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD, default 2); otherwise traffic falls back to Gateway V1 at the next refresh boundary. No mid-flight fallback. Caveats: - Probe wiring is skipped entirely for Direct mode and when HTTP/2 is not configured; controlled by RxDocumentClientImpl.useThinClient. - QueryPlan, metadata reads, and AllVersionsAndDeletes change feed continue to route through Compute Gateway (Gateway V1). - Probe failures are absorbed inside the orchestrator and the trigger is fire-and-forget on the GEM scheduler, so probe issues can never trip CosmosClient initialization or fail a topology refresh. - EndpointOrchestrator implements Closeable and is closed from GlobalEndpointManager.close() so no further probes are issued after client shutdown. THINCLIENT_ENABLED now defaults to true; opt out via COSMOS.THINCLIENT_ENABLED=false or COSMOS.THINCLIENT_PROBE_ENABLED=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: probe single-flight, body-drain lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 second-batch review feedback - LocationCache.getThinClientRegionalEndpoints now walks both read and write region endpoint maps so single-master write-region failures still flip the probe gate. - EndpointOrchestrator.forceUnhealthy(reason) provides a non-HTTP path to flip the gate; GlobalEndpointManager calls it when topology says thin-client is eligible but no regional endpoint resolves. - Symmetric hysteresis: new COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD (default 1) so operators can require N consecutive GREEN cycles before flipping back to proxy. - Extracted RxDocumentClientImpl.useThinClientStoreModel(...) body into package-private static shouldUseThinClientStoreModel for direct unit testability; added ThinClientRoutingGateTests covering 9 routing paths. - EndpointOrchestratorTests.stubResponse now returns Mono.empty() to avoid Unpooled.EMPTY_BUFFER refCnt underflow across multiple probe calls. - Removed unused locals; added recoveryThresholdRequiresMultipleGreenCycles, forceUnhealthy_flipsGateToRedWithoutRunningProbe, forceUnhealthy_onClosedOrchestrator_isNoOp tests. All 57 unit tests in the touched files pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify CHANGELOG: probe recovery threshold is now configurable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review (rounds 4-8): reactor-chain probe, FQN cleanup, fix gwV2Cto and ThinClient user-agent assertions - GlobalEndpointManager: convert thin-client probe trigger to a Mono<Void> chained into the topology-refresh reactor pipeline (replaces fire-and-forget subscribe). Removes thinClientProbeDisposable field and its close() handling since cancellation now propagates through the outer subscription. - EndpointProbeClient/EndpointProbeClientTests/ThinClientProbeWiringTests: replace inline FQNs with imports (java.io.Closeable, java.util.List, java.net.ConnectException, com.azure.cosmos.implementation.http.HttpHeaders). - ClientConfigDiagnosticsTest: compute gwV2Cto dynamically from Configs.isThinClientEnabled() so assertions remain valid after the default flip to true. - ConfigsTests: update default-threshold assertions from 2 to 1 to match DEFAULT_THINCLIENT_PROBE_FAILURE_THRESHOLD=1. - UserAgentContainerTest.UserAgentIntegration: expect '|F4' suffix because the ThinClient feature flag (1 << 2) is now included by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test fallout from default ThinClient enablement UserAgentSuffixTest.validateUserAgentSuffix and CosmosDiagnosticsTest.generateHttp2OptedInUserAgentIfRequired: include UserAgentFeatureFlags.ThinClient in computed |F<hex> suffix when COSMOS.THINCLIENT_ENABLED is true (now default after Gateway V2 default enablement). Mirrors RxDocumentClientImpl.addUserAgentSuffix + UserAgentContainer.setFeatureEnabledFlagsAsSuffix behavior. SinglePartitionDocumentQueryTest.querySinglePartitionDocuments: spy on both gateway-proxy and thin-proxy and assert exactly one invocation. Previous code only spied on the proxy implied by useThinClient() config intent, which races with the probe-healthy gate -- routing AND's intent with isProxyProbeHealthy() so on first cycle the request may go through gateway even when thin-client is configured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe in Http2PingKeepaliveTest The test installs an iptables DROP on thin-client port 10250 to verify that Http2PingHandler closes the broken connection after consecutive PING ACK timeouts and the recovery request uses a new connection on the same regional endpoint. After default Gateway V2 enablement, the connectivity probe also fires HTTP/2 POSTs to port 10250 on every account refresh. With iptables dropping that port, the probe trips proxyHealthy=false, useThinClient StoreModel() returns false, and the data plane request routes through Gateway V1 on port 443 -- which iptables is not dropping. Result: the PING handler never fires, the warm-up and recovery requests use the same gateway channel, and the assertion 'recovery channel must differ from initial' fails (both ended up as 77af2e47 on build 6419227). Set COSMOS.THINCLIENT_PROBE_ENABLED=false in beforeClass so the probe short-circuits to a no-op, EndpointProbeClient.proxyHealthy stays optimistically true, and the data plane request actually flows over port 10250 where the iptables DROP can take effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe by default for E2E tests in TestSuiteBase Build 6424287 surfaced two new failure patterns: 1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures) - asserts substatus 1003 from the thin-client routing path, but observed 0 because the data plane was routed to Gateway V1 instead of the proxy. 2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures) - TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained ':10250/', i.e. nothing actually went to the thin-client proxy. Both patterns trace to the same source: the new connectivity probe is enabled by default, the proxy-side /connectivity-probe endpoint is not deployed in every CI test account yet, and the default failure threshold is 1. So after the first probe cycle the SDK marks the proxy unhealthy and routes data plane traffic to Gateway V1, which breaks tests that explicitly assert thin-client routing. Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic, configuration-driven routing. Tests that exercise the probe itself (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are not affected. Also drop the now-redundant per-class override in Http2PingKeepaliveTest - the base class disables it, and the test's @AfterClass clear would otherwise re-enable the probe for any subsequent E2E test sharing the JVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Disable thin-client probe by default for E2E tests in TestSuiteBase" This reverts commit ad9e3dff1db97df0d53a6f92aaec709676caff19. * Add per-class thinclient probe disable in CosmosNotFoundTests and PerPartitionCircuitBreakerE2ETests Companion to the prior revert. The revert undid the global TestSuiteBase probe disable (which masked production behaviour). This commit adds the necessary per-class disable to the two test classes whose assertions explicitly require thinclient routing: CosmosNotFoundTests (thinclient group) and PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group). Both clear the property in their @AfterClass. Http2PingKeepaliveTest already has its own disable (restored by the revert). Production callers continue to get the connectivity probe ON by default with the production failure threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add unit tests for QueryPlan and stored-procedure thin-client routing Covers 5 new scenarios in ThinClientRoutingGateTests: - ExecuteStoredProcedure on a StoredProcedure resource routes to thin client - Non-execute StoredProcedure ops (Create) route to Gateway V1 - OperationType.QueryPlan routes to thin client - QueryPlan returns false when probe is unhealthy - ExecuteStoredProcedure returns false when probe is unhealthy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove endpoint-probe content from QueryPlan PR branch Reverts the merge of jeet1995/thin-client-probe-flow that was brought into this PR earlier, while keeping the branch up-to-date with upstream/main. Net diff vs upstream/main is now only the QueryPlan and StoredProcedure Gateway V2 routing changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * review: address PR #49437 items 3 and 4 Item 3: remove unused COSMOS.USE_AAD_AUTH system property from the thinclient profile in azure-cosmos-tests/pom.xml. The Maven property cosmos.use.aad.auth is not defined anywhere in the repo, so the substitution produced a literal that Boolean.parseBoolean reads as false; TestSuiteBase already falls back to the COSMOS_USE_AAD_AUTH env var. Item 4: in Http2PingKeepaliveTest, gate the THINCLIENT_PROBE_ENABLED=false override behind !THIN_CLIENT_ENABLED so the probe is disabled only in the pure Compute Gateway (Gateway V2) branch on port 443, where thin-client routing is off entirely and probe POSTs to port 10250 are pure noise. In the thin-client branch (port 10250) the probe is intentionally left enabled so the same production code path that gates thin-client routing is exercised. Mirror the conditional in afterClass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 Review 2 feedback - QueryPlanRetriever: clarify thin-client / Gateway V1 routing comment; explain that Gateway V1 is pinned only when PartitionKeyDefinition is unavailable to convert proxy queryRanges to EPK hex ranges. - RxDocumentClientImpl + DocumentQueryExecutionContextFactory: plumb the resolved DocumentCollection into validateCustomQueryForReadManyByPartitionKeys -> fetchQueryPlanForValidation so the validation query-plan request is thin-client-eligible instead of forcibly pinned to Gateway V1. - EndpointProbeClient: extract DiagnosticsSnapshot and ProbeResult into top-level types (EndpointProbeDiagnosticsSnapshot, EndpointProbeResult); remove @SuppressWarnings("unused"); use the failure reason in RED-cycle logs and toString(); verbose NPE message in the constructor naming the required dependency and wiring entry point. - CHANGELOG: move thin-client entry under Features Added; one-line summary with HTTP/2 + gatewayMode requirement and PR link. - CosmosNotFoundTests: add forceHttp1IfGatewayMode helper applied to every fast-group client construction so Gateway-mode clients in the fast group use HTTP/1.1 and are guaranteed not to route through the proxy. Keep the COSMOS.THINCLIENT_PROBE_ENABLED=false set/clear for the thinclient group's defensive teardown. - PerPartitionCircuitBreakerE2ETests: drop the THINCLIENT_PROBE_ENABLED property mutation now that probe behavior is asserted only by tests that opt in explicitly. * Fix CI compile: import ConnectionPolicy in CosmosNotFoundTests * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Thread DocumentCollection into readMany query-plan validation readManyByPartitionKeys validates any caller-supplied custom query by fetching a query plan and asserting it is single-partition and non-hybrid. Until now that validation called fetchQueryPlanForValidation with no DocumentCollection, so the plan request was pinned to Gateway V1 (useGatewayMode = (partitionKeyDefinition == null)). Thread the container's DocumentCollection from RxDocumentClientImpl.validateCustomQueryForReadManyByPartitionKeys -> DocumentQueryExecutionContextFactory.fetchQueryPlanForValidation so QueryPlanRetriever has the PartitionKeyDefinition needed to convert PartitionKeyInternal-formatted queryRanges from the thin client (Gateway V2) proxy into the EPK-hex Range<String> entries the query pipeline consumes. With this wiring, the validation query plan goes to the thin client when the client is configured for it, and remains on Gateway V1 otherwise. No behavior change on the non-thin-client path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): disable probe in CI fault-injection tests + honor kill switch in forceUnhealthy Stage 1990 of build 6432767 had 83 failures in ClientRetryPolicyE2ETestsWithGatewayV2#serviceUnavailableWithGatewayV2 because the probe gate added by PR #49437 flips routing from :10250 to :443 when CI test accounts lack /connectivity-probe (failureThreshold=1 trips proxyHealthy=false on the first failed probe and the assertion at TestSuiteBase#assertThinClientEndpointUsed then sees :443 and fails). Test fix (per CosmosNotFoundTests precedent, commit 3b392e3acf5): - ClientRetryPolicyE2ETestsWithGatewayV2: disable probe in @BeforeClass / clear in @AfterClass. - ThinClientTestBase: same lifecycle in enableThinClientForTest / clearThinClientForTest so all ThinClient*E2ETest subclasses inherit the guard. Source fix: honor COSMOS.THINCLIENT_PROBE_ENABLED kill switch inside EndpointProbeClient.forceUnhealthy(), closing the item-5 invariant gap (previously only runProbeCycle() checked the flag, so a mismatch-driven forceUnhealthy could mutate proxyHealthy even with the probe disabled). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): kill switch bypasses probe-health gate When COSMOS.THINCLIENT_PROBE_ENABLED=false, no probe cycles run, so the probe-health signal is stale/meaningless. Treat probe as healthy in that case so the gate is driven solely by COSMOS.THINCLIENT_ENABLED (already folded into useThinClient at construction). This fixes regressions in CI test classes (GatewayReadConsistencyStrategyE2ETest, FaultInjectionWithAvailabilityStrategyTestsBase, ThinClientQueryE2ETest) where probe to /connectivity-probe fails on test accounts and flips routing to :443 instead of :10250. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(thin-client): disable probe in CI fault-injection / failover / consistency tests Adds COSMOS.THINCLIENT_PROBE_ENABLED=false in @BeforeClass (and clears in @AfterClass) for PerPartitionCircuitBreakerE2ETests, PerPartitionAutomaticFailoverE2ETests, FaultInjectionServerErrorRuleOnGatewayV2Tests, and GatewayReadConsistencyStrategyE2ETest. These tests run against CI accounts whose /connectivity-probe endpoint is not deployed. Without this, the first failed probe trips proxyHealthy=false and routing falls back to Gateway V1 (port 443), causing assertThinClientEndpointUsed (which expects port 10250) to fail. Locally validated PPAF (16 tests, 0 failures), GRCS (3 tests, 0 failures), FI (3 tests, 0 failures). PPCB still in progress at commit time; pushing to let local + CI run in parallel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client QueryPlan error returning statusCode 0 instead of 400 The thin-client/Gateway-V2 proxy returns a non-2xx response with a raw, non-JSON, NUL-padded error body for invalid-syntax queries. In RxGatewayStoreModel.validateOrThrow, new CosmosError(body) attempted to parse that body as JSON and threw IllegalArgumentException, which escaped the method before the existing status-carrying throw could run. Upstream then wrapped it as statusCode 0. Wrap the CosmosError(body) construction in a narrow try/catch (IllegalArgumentException) and fall back to the non-parsing CosmosError(errorCode, message) constructor with a sanitized body. The existing throw now fires with the correct status (400) and the proxy error text is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden ThinClientQueryE2ETest assertions (F1-F5) F1: strict 400 + unconditional thin-client endpoint check on invalid query, locking the statusCode-0->400 fix. F2: ordered-vs-sorted-set document-ID comparison (ORDER BY sequence-compared, others set-compared). F3: ID-set equality + no-duplicate check across drained continuation pages. F4: numeric-tolerance (1e-6) comparison for scalar and GROUP BY aggregates to avoid float-formatting false mismatches. F5: validated vector/full-text/hybrid queries match Direct vs thin-client end-to-end through the proxy. Validated live via -Pthinclient: 84 tests, 0 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace thin-client test matrix with reverse-engineered E2E test spec Rewrite THINCLIENT_TEST_MATRIX.md as a reviewable test-design specification reverse-engineered from the committed ThinClientQueryE2ETest code (84 tests). Documents the differential-testing oracle (Direct :443 baseline vs thin client :10250 SUT), the data-model fixture, every assertion contract (endpoint provenance, ordered-vs-unordered ID equality, scalar/GROUP BY tolerance), the full 84-test matrix, the F1-F5 hardened special cases (continuation draining, invalid-query 400, vector/FTS/hybrid ranking, readMany validation path), advertised-feature coverage, and known gaps (CountIf/DCount/MultipleOrderBy) for reviewer sign-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply Aditya Sarpotdar review feedback to thin-client E2E tests Executes actionable review feedback on ThinClientQueryE2ETest plus two harness fixes surfaced during live validation against thin-client-multi-region-ci. ThinClientQueryE2ETest: - Strict ordering: ORDER BY results validated with isStrictlyOrdered across the board (stricter-by-default, per reviewer guidance) instead of set parity. - Add testMultipleOrderBy() with a composite-index container to cover the MultipleOrderBy query feature. - Add testDCount() using the canonical Cosmos DCount idiom (COUNT over a DISTINCT VALUE subquery); SQL-standard COUNT(DISTINCT ...) is not valid Cosmos SQL grammar. - Reword multi-EPK-range comment/javadoc to reflect emulator/backend reality (multiple metadata ranges served by a single backend partition; SDK routing and query pipeline still exercised). Harness fixes: - TestSuiteBase: add wait-and-poll utility waitForCollectionToBeAvailableToRead (predicate on NotFound/substatus 1013) to deflake "Collection is not yet available for read" on freshly created containers; update call sites in OrderbyDocumentQueryTest, NonStreamingOrderByQueryVectorSearchTest, QueryValidationTests, ReadFeedCollectionsTest. - SinglePartitionDocumentQueryTest: make the processMessage Mockito assertion mode-aware (thin-client routes query plan + query => times(2)). Validated live against thin-client-multi-region-ci: 86/86 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Redesign thin-client probe to per-region probe-and-cache model Convert the global circuit-breaker (failure/recovery hysteresis re-probing all regions every refresh) into a per-region one-shot probe-and-cache: probe only the delta of unproven regions, cache successes across account refreshes, and never re-probe a proven region. The routing gate stays conservative (Gateway V1 until every known region is proven, Gateway V2 once all succeed). - Configs: drop failure/recovery threshold knobs; add THINCLIENT_PROBE_MAX_RETRIES (default 3) and isThinClientExplicitlyEnabled() bypass. - EndpointProbeClient: per-region cache, delta probing, retry-N, all-known-succeeded gate, forced-unhealthy latch; per-probe timeout from thin-client connection timeout. - EndpointProbeDiagnosticsSnapshot: expose knownRegionCount/succeededRegionCount. - RxDocumentClientImpl: gate probe wiring/routing on !isThinClientExplicitlyEnabled. - Tests: rewrite EndpointProbeClientTests; update ConfigsTests, ThinClientProbeWiringTests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: opt-in thin-client QueryPlan routing + kill-switch, stop advertising CountIf, remove PR-reference comments * Address review feedback: broaden gateway error fallback to ClassCastException, drop response ObjectNode mutation, fail-fast on empty queryRanges, align readMany routing test with opt-in * Make thin-client QueryPlan no-EPK-headers contract explicit in wrapInHttpRequest * Fix missing StandardCharsets import in RxGatewayStoreModelTest * Test: add QueryOracle-derived LIKE/scalar-expression thin-client parity tests; drain only first page in OperationPolicies readAllItems to avoid full-container enumeration timeout * Test: verify a cached proxy-generated query plan still executes correctly through the thin client * Test: cross-partition tests use a dedicated multi-physical-partition fixture and assert >1 partition key range contacted; enforce ordered full-row comparison when id is not projected; clearer naming * Add CHANGELOG entry for QueryPlan request routing to Gateway V2 * Call out Execute Stored Procedure support in CHANGELOG entry * Fix flaky region/timing-sensitive fault-injection E2E tests CustomerWorkflowPartitionLevelCircuitBreakerTest and CustomerWorkflowHighE2ETimeoutTest: bump the ThresholdBasedAvailabilityStrategy threshold from 100ms to 300ms so the cold primary-region connection can dispatch the request and record the injected fault before the availability-strategy hedge to a warm secondary region wins the race and cancels it (which previously yielded 0 fault hits). PerPartitionCircuitBreakerE2ETests: before asserting short-circuit routing, wait until the first preferred region is actually marked Unavailable, removing the race between the failure-counter estimate and the asynchronous routing exclusion. Scoped to single-faulty-region configs (expectedRegionCountWithFailures == 1) and run once so fault-in-all-regions configs (which can never short-circuit) do not hit the method timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove test AAD auth toggle; move thin-client test matrix to docs/ and align with ThinClientQueryE2ETest * Thin-client review: tri-state enablement/probe-decision + EndpointProbeClient cleanup - Configs: COSMOS.THINCLIENT_ENABLED is tri-state (null=unset default-on, TRUE=opt-in, FALSE=opt-out); consumers interpret null. Remove probe kill switch (COSMOS.THINCLIENT_PROBE_ENABLED) and probe-path config (path is fixed /connectivity-probe). - ThinClientConnectivityConfig (new): single authority for enablement + the shouldUseThinClientStoreModel routing gate, taking the nullable probe decision so null never collapses into a boolean clause at the call site. - GlobalEndpointManager.getProxyProbeDecision(): tri-state Boolean (null = no decision: probe not wired). - EndpointProbeClient: hard-code PROBE_PATH; remove cycleId correlation and unused EndpointProbeResult.reason; crispen runProbeCycle (single currentGate no-op Mono, removeIf delta). - Tests: repoint routing-gate tests to the 5-arg tri-state signature (+null/opt-in cases); remove THINCLIENT_PROBE_ENABLED usage from E2E tests; drop removed-config tests. * Consolidate probe state into single source-of-truth map; remove force-unhealthy latch and per-cycle copies - Replace volatile knownEndpoints Set + probeSucceeded CHM with one ConcurrentHashMap<URI,Boolean> (probeEndpointToProbeSuccessState): keys = current topology, value = proven/not-yet-proven. Gate is non-empty AND all TRUE. - Remove the force-unhealthy latch: an empty/null topology now empties the map via retainAll so the gate falls to false naturally (route to Gateway V1). - runProbeCycle reconciles and probes by iterating the provided Collection<URI> directly (reference only, no LinkedHashSet copies) and drops the delta/empty short-circuit fast paths; the Flux filters not-yet-proven endpoints inline. - applyCycleResult uses replace() so a region pruned by a concurrent reconcile is never resurrected. Unit suite green: 2579 tests, 0 failures. * Add cross-partition aggregate / GROUP BY parity tests for thin-client query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix EPK over-span; add QueryPlan parity tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix partition key EPK over-span For a partial (prefix) hierarchical (MULTI_HASH) partition key, the thin-client (Gateway V2) store model previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter. The proxy therefore resolved the request to the owning physical partition and returned every co-located document (an over-span) instead of only those matching the prefix. ThinClientStoreModel.wrapInHttpRequest now detects a prefix MULTI_HASH key and sets READ_FEED_KEY_TYPE=EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from(), so RntbdRequestHeaders serializes the prefix EPK sub-range [hash(prefix), hash(prefix) + "FF") as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. QueryPlan requests and full (non-prefix) keys are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client query E2E parity regression tests Ports ThinClientQueryE2ETest and its ThinClientTestBase helper. The suite self-baselines every query against a Direct (RNTBD) client and the thin-client (:10250 HTTP/2) path, asserting identical results. This directly guards the MULTI_HASH prefix partition-key EPK over-span fix: prefix (single-component) hierarchical-partition-key queries must return only the narrow matching set, matching Direct, rather than over-spanning to all co-located documents. Both files are runtime self-enabling (enableThinClientForTest() + builder-driven HTTP/2 via setEnabled(true)); no module property or TestSuiteBase changes needed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reuse prefix EPK range on thin-client parallel prefix-query path The earlier fix (499f928) only scoped the prefix MULTI_HASH read when the request carried a non-null partition key. The parallel prefix-query path intentionally nulls the partial hierarchical partition key and instead carries the narrow prefix effective-partition-key range on the request's feedRange, so that guard never fired there and the thin-client read still over-spanned to the whole physical partition. Mark such requests (prefixPartitionKeyQuery) in ParallelDocumentQueryExecutionContextBase and, in ThinClientStoreModel, reuse the already-computed FeedRangeEpkImpl range as the doc-level EPK filter (START_EPK/END_EPK + StartEpkHash/EndEpkHash) instead of recomputing getEPKRangeForPrefixPartitionKey from a partition key we no longer have. This mirrors the Direct/GatewayV1 decision points and honors DRY/SRP. Also relax assertThinClientEndpointUsed so an all-QueryPlan diagnostics context passes: in thin-client mode QueryPlan calls resolve via the classic gateway and are the only requests permitted on a non-thin-client endpoint; any non-QueryPlan (data) request on a non-thin-client endpoint still fails the assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client prefix EPK RNTBD headers, propagate prefix flag on clone, add readAllItems/readMany prefix HPK tests ThinClientStoreModel.wrapInHttpRequest now sets all five EPK-specific RNTBD headers (ReadFeedKeyType, StartEpk/EndEpk, StartEpkHash/EndEpkHash) in a single block directly on the RNTBD request, replacing the split request-header-map + post-construction approach. Wire encoding is unchanged (hex-string UTF-8 bytes for StartEpk/EndEpk, decoded hash bytes for the hash headers). RxDocumentServiceRequest.clone() now copies prefixPartitionKeyQuery so hedged/availability-strategy/retry clones keep the prefix signal and do not over-span. ThinClientQueryE2ETest adds prefix-HPK regression coverage for readAllItems (ReadFeed) and readManyByPartitionKeys, asserting Direct-vs-thin-client parity and per-tenant scoping (no over-span). * Simplify prefix-query flag assignment in ParallelDocumentQueryExecutionContextBase Assign isPrefixPartitionKeyQuery directly from isPartialPartitionKeyQuery(...) and guard only the PARTITION_KEY-setting branch with !isPrefixPartitionKeyQuery, removing the if/else. Behavior is unchanged: a partial (prefix) key still leaves partitionKeyInternal null so the prefix FeedRangeEpkImpl survives createDocumentServiceRequestWithFeedRange. * Add CHANGELOG entry for thin-client prefix hierarchical partition key over-span fix * Minimize prefix-query flag change vs main in ParallelDocumentQueryExecutionContextBase Keep main's single 'if (!isPartialPartitionKeyQuery)' structure; only capture the predicate in isPrefixPartitionKeyQuery and pass it to request.setPrefixPartitionKeyQuery. Removes the if/else. * Order thin-client prefix detection to prefer the cached feed-range path In ThinClientStoreModel.wrapInHttpRequest, check the cached prefix feed-range path (isPrefixPartitionKeyQuery + FeedRangeEpkImpl) first and fall back to recomputing from the partition key only for the PK-bearing prefix case. Behavior is unchanged (the two cases are mutually exclusive). * Address Copilot review: validate all requests in thin-client endpoint assertion TestSuiteBase.assertThinClientEndpointUsed now validates every request instead of early-returning on the first thin-client match, so a mixed scenario (some data requests via the classic gateway) fails; QueryPlan requests remain exempt and null endpoints are handled. ThinClientTestBase.assertThinClientEndpointUsed delegates to the shared TestSuiteBase implementation, removing duplicated logic that could NPE on a null endpoint and reject valid QueryPlan-via-gateway scenarios. * Simplify thin-client prefix EPK handling to reuse HTTP EPK headers Replace the explicit MULTI_HASH prefix-detection and pre-serialization range computation with a header-presence branch: when the request already carries START_EPK/END_EPK HTTP headers (populated upstream by FeedRangeEpkImpl for prefix hierarchical partition keys and explicit EPK-range reads), set only the additive StartEpkHash/EndEpkHash RNTBD tokens. The ReadFeedKeyType/StartEpk/ EndEpk string tokens are copied verbatim from those HTTP headers by RntbdRequestHeaders#addStartAndEndKeys during RntbdRequest.from(), so they no longer need to be set here. Removes now-unused imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove dead prefix-query flag and expand HPK thin-client test coverage Following the simplification of the prefix EPK fix to reuse the existing StartEpk/EndEpk HTTP headers (a56db9ad44c), the RxDocumentServiceRequest prefixPartitionKeyQuery flag is no longer read anywhere. Remove the field, its accessors, the clone copy, and the caller assignment in ParallelDocumentQueryExecutionContextBase. Test changes: - Retrofit CosmosMultiHashTest into the thinclient TestNG group so the MULTI_HASH prefix over-span coverage also runs against GatewayV2 (proxy :10250) over HTTP/2, not just the emulator gateway. - ThinClientQueryE2ETest: correct the prefix-scope Javadoc to describe the actual header-copy behavior and add a deterministic prefix-partition-key readAllItems regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert ParallelDocumentQueryExecutionContextBase to main The prefix-query flag on RxDocumentServiceRequest was removed in 03a7c672b95 after the fix was simplified to reuse the StartEpk/EndEpk HTTP headers. The only functional edit in this file (the setter call) went away with it, leaving behind unnecessary refactor noise. Restore the file to its upstream/main state so the hotfix touches no query execution code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore thin-client-probe symbols dropped during merge of #47759 The merge of AzCosmos_GatewayV2_QueryPlanSupport (#47759) dropped probe's thin-client-probe supporting code while keeping the method bodies that reference it, breaking compilation: - GlobalEndpointManager: restore imports (http.HttpClient, java.util.Set) and the thinClientProbeClient AtomicReference field. - LocationCache: restore getThinClientRegionalEndpoints() and the collectThinClientEndpointsAllOrNothing() helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos): allowlist subStatus 1003 in thinclient readiness-probe retry The @BeforeSuite container-readiness probe issues a QueryPlan against a freshly created container. With thin-client default-on, a fresh container in a multi-region account can transiently return 404/subStatus 1003 (OWNER_RESOURCE_NOT_EXISTS) during regional metadata propagation. The classifier did not allowlist 1003, so the probe treated it as non-retryable and cascade-skipped the entire thinclient suite. Add OWNER_RESOURCE_NOT_EXISTS to the NOTFOUND retryable allowlist in isRetryableCollectionReadinessFailure so the probe retries through propagation and the suite runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos-benchmark): parameterize SDK version + add RunId/EndpointFlavor cold-start knobs Instrument azure-cosmos-benchmark for the cold-start Create-latency matrix driving the thin-client probe (PR #49437): - pom: parameterize azure-cosmos + azure-cosmos-encryption versions (-Dazure-cosmos.version=...) so GA 4.79/4.80/4.81 and the probe build can be benchmarked from one harness. - Common tags (RunId, SdkVersion, ConnectionMode, EndpointFlavor, Phase) applied to every metric so a shared Grafana dashboard can slice runs. - endpointFlavor knob (ComputeGateway|ThinClient) maps to COSMOS.THINCLIENT_ENABLED at client build to drill both the Gateway V1 and Gateway V2 (thin-client) HTTP/2 paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce QueryPlan thin-client routing invariant and harden container-readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add kill-switch flip validation for QueryPlan thin-client routing Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore isAnyRegionShortCircuitedForPartition helper dropped during #47759 merge The #47759 merge conflict resolution kept HEAD's short-circuit-wait loop (which calls isAnyRegionShortCircuitedForPartition) but dropped the method definition. Recovered from probe tip 42c2c7d733f and re-inserted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove azure-cosmos-benchmark and HTTP/2 ping-handler changes (moved to #49725) These changes belong in PR #49725, not the endpoint-probe PR #49437. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Run thin-client probe cycle on every account refresh (delta-gated) Wire runThinClientProbeCycleMono() into the account-refresh funnel in GlobalEndpointManager so the connectivity probe runs on every refresh, delta-gated to new/unproven regions in EndpointProbeClient. Previously the probe driver had no production callers, leaving the default-on thin-client routing gate permanently false. Add a GEM delta-refresh unit test proving a subsequent refresh probes only newly-added regions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client endpoint-probe (implicit path) E2E test lane + GEM probe wiring Wire runThinClientProbeCycleMono() into GlobalEndpointManager.refreshLocationPrivateAsync so the connectivity probe runs on every account refresh: as a prefix when a fresh databaseAccount is present, and after the foreground database-account read (.map -> .flatMap). Add an implicit-path test lane that leaves COSMOS.THINCLIENT_ENABLED unset (thin-client default-on, routing gated on the probe verdict with Gateway V1 fallback) while keeping HTTP2_ENABLED true: - new 'thinclient-endpoint-probe' maven profile (azure-cosmos-tests/pom.xml) - new Cosmos_Live_Test_ThinClient_EndpointProbe CI stage (tests.yml) - live-thinclient-endpoint-probe-platform-matrix.json - thinclient-endpoint-probe-testng.xml suite - register thinclientEndpointProbe group in TestSuiteBase before/after suite - E2E probe tests: point-operation, query, change-feed, stored-procedure + shared base Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align query/* and test files with Azure/main to reduce PR diff Reset QueryPlanRetriever, PartitionedQueryExecutionInfo and 9 test files to Azure/main. These carried Gateway V2 QueryPlan work (partitionKeyDefinition guard, CountIf, queryRanges removal) and parity-test adaptations that overlap upstream #47759, which Azure/main already owns. Scoping this PR to the thin-client connectivity-probe flow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove isThinClientEnabled() helper; require positive probe verdict to route thin-client Collapse all call sites onto the tri-state reader Configs.isThinClientEnabledExplicitly() and remove the redefined isThinClientEnabled() helper, which had flipped the unset default from false to true. Thin-client routing now requires an explicit opt-in or an affirmative connectivity-probe verdict; an unset flag no longer routes to Gateway V2 by default (null/FALSE probe verdict -> Gateway V1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Collapse thin-client enablement to single nullable isThinClientEnabled() Rename Configs.isThinClientEnabledExplicitly() to isThinClientEnabled() (nullable Boolean) and remove the hasUserExplicitlyEnabledThinClient() and ThinClientConnectivityConfig.isExplicitThinClientOptIn() helpers. The routing gate shouldUseThinClientStoreModel now takes the tri-state Boolean directly: a non-null value is a hard contract (TRUE routes to Gateway V2, FALSE pins to V1), and null falls back to the connectivity-probe verdict. This eliminates the scattered Boolean.equals opt-in checks. Default remains null (not default-on). Updated ThinClientRoutingGateTests implicit-path cases to pass null for the tri-state arg (probe-gated) instead of false. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client probe E2E tests into shared classes via dual TestNG group Tag the existing ThinClient*E2ETest classes (and ThinClientTestBase) with both the `thinclient` and `thinclientEndpointProbe` groups so the same test bodies run in either CI lane against a different ambient enablement path: - thinclient lane: -DCOSMOS.THINCLIENT_ENABLED=true (explicit opt-in) - thinclient-endpoint-probe lane: flag unset -> endpoint connectivity probe drives routing Delete the 5 duplicate ThinClientEndpointProbe* copies; the probe suite XML discovers by group across com.azure.cosmos.*, so it auto-picks up the retagged classes with zero duplication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thinclientEndpointProbe group to shared suite setup/teardown The consolidated thin-client E2E tests are dual-tagged with the thinclientEndpointProbe group, but the shared @BeforeSuite/@AfterSuite in TestSuiteBase (which creates and deletes the SHARED_* collections) was still gated only on the thinclient group. Under the probe lane's group filter the suite setup never ran, leaving the shared collections null and causing an NPE in cleanUpContainer during @BeforeClass. Adding the group makes both lanes green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix ClientConfigDiagnosticsTest stale gwV2Cto assertions Update the gw connCfg assertions to expect gwV2Cto:PT5S instead of gwV2Cto:n/a, matching the intentional diagnostics change where the thin-client (gateway V2) connect timeout is emitted by default when COSMOS.THINCLIENT_ENABLED is unset. This also resolves the cascading sessionRetryOptionsInDiagnostics failure caused by full() leaking a system property when its assertion threw before cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Update UserAgent tests to expect |F4 ThinClient feature flag ThinClient is default-enabled when COSMOS.THINCLIENT_ENABLED is unset on this branch, so the UserAgent suffix now carries the |F4 feature flag. Update the stale assertions to mirror RxDocumentClientImpl.addUserAgentSuffix dynamically (ThinClient default-on + Http2/Http2PingHealth) instead of hardcoding the suffix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align endpoint probe cache to .NET add-only semantics Refactor EndpointProbeClient to an add-only proven-healthy endpoint cache, matching the .NET implementation (PR azure-cosmos-dotnet-v3#5970). Once an endpoint is proven ThinClient-routable it is never pruned; the routing gate is evaluated against a volatile snapshot of the current topology so a transiently vanished proven region does not gate current routing and is not re-probed when it re-appears. Probe cycles only probe the delta of new, not-yet-proven endpoints on each account refresh. Add unit tests covering vanish/re-appear and unsupported-new-region scenarios (region added without support blocks the gate; removing it restores routing). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix CosmosDiagnosticsTest for default-on ThinClient |F4 user-agent suffix Fold ThinClient into generateHttp2OptedInUserAgentIfRequired so the helper dynamically mirrors production feature-flag suffixing, and remove a double-wrap of directClientUserAgent (already suffixed at construction). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Gate thin-client endpoint assertion on explicit opt-in only On the implicit/probe path (COSMOS.THINCLIENT_ENABLED unset), thin-client routing depends on the endpoint probe greenlighting all current regions. Under fault injection the probe is not guaranteed to converge, so requests may legitimately fall back to Gateway V1 (:443). Assert thin-client routing only when thin-client is explicitly opted in (THINCLIENT_ENABLED=true), the sole config where routing is deterministic because the probe is bypassed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: tri-state THINCLIENT_ENABLED parsing, probe-client close, fire-and-forget probe cycle, single-flight tests - Configs: parse COSMOS.THINCLIENT_ENABLED via explicit true/false whitelist (parseTriStateThinClientEnabled); any other value logs a warning and is treated as unset (null -> probe-gated) instead of Boolean.parseBoolean silently collapsing every non-"true" string to a hard opt-out. - GlobalEndpointManager: close() now cancels the in-flight probe-cycle subscription and closes the EndpointProbeClient so a stale cycle drops its result. Probe cycle is fired fire-and-forget on force-refresh and topology refresh so init()/cross-region retry never block on the probe; routing stays on Gateway V1 until the probe proves the proxy endpoints. - Tests: add ConfigsTests.thinClientEnabledInvalidValueTreatedAsUnset and two EndpointProbeClientTests covering single-flight overlap skip and closed-mid-cycle result-drop (gate stays conservative). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refactoring. * Fix SinglePartitionDocumentQueryTest stale thin-client routing predicate useThinClient() now reflects config-eligibility only; the test must gate on actual per-request routing (read locations + probe/opt-in) to match where the query lands. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR review: consolidate probe-wiring gate on canThinClientBeUsed() - Gate probe wiring in RxDocumentClientImpl on canThinClientBeUsed() so an explicit THINCLIENT_ENABLED=true still wires the probe (routing bypasses it), covering true->null runtime transitions. Removed dead canThinClientBeImplicitlyEnabled() from ThinClientConnectivityConfig and updated javadoc/comments/pom p…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines