[BUG] Cosmos Spark connector can hang indefinitely during change-feed offset discovery when DNS resolution fails

[joeyp-openai]

Affected component: com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12

Problem

The Azure Cosmos DB Spark connector can block indefinitely while discovering change-feed offsets when a Cosmos endpoint cannot be resolved. Instead of failing the streaming query, the connector remains stuck inside a metadata request, preventing Spark and external orchestrators from detecting the failure or retrying.

We encountered this when an Azure-managed private DNS zone group failed to create several regional Cosmos endpoint records. Consequently, requests to the affected regional endpoint produced UnknownHostException. Azure documents both the role of private DNS zone groups in maintaining regional records and the relationship between UnknownHostException and unresolved Cosmos endpoints. [Azure Private Link documentation](https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-configure-private-endpoints), [Cosmos Java SDK troubleshooting](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/troubleshoot-java-sdk-v4).

The underlying DNS failure is a separate infrastructure issue. This report concerns the connector’s failure to terminate when that condition persists. (The Azure ticket related to the infra issue is 2607310010001458)

Actual behavior

1. A Structured Streaming query using cosmos.oltp.changeFeed starts successfully.
2. The first microbatch invokes ChangeFeedMicroBatchStream.initialOffset() or latestOffset().
3. Feed-range or partition-metadata discovery enters an untimed Reactor Mono.block() call.
4. DNS failures are retried inside the reactive subscription, which may never complete or return control to the connector’s outer retry loop.
5. No microbatch completes, no progress event is emitted, and the streaming query never reports a failure.

In our production incident, applications remained stuck for approximately 2 hours and 45 minutes, until the scheduler forced termination.

The outer transient-error retry policy also defaults to effectively unlimited attempts, but bounding that policy alone does not solve the problem: an in-flight Mono.block() can hang indefinitely without ever returning to the retry loop.

Expected behavior

If change-feed metadata discovery cannot complete within a configurable deadline, the connector should:

• Cancel the in-flight metadata request.
• Fail the streaming query with an actionable timeout or underlying connection error.
• Allow Spark and the external orchestrator to retry or reschedule the workload.
• Preserve normal transient-error retries when they complete within the configured deadline.

Reproduction

1. Configure a Cosmos DB account with a private endpoint.
2. Make a required regional Cosmos DNS record unavailable, or otherwise cause the selected regional endpoint to consistently return UnknownHostException.
3. Start a Spark Structured Streaming query using cosmos.oltp.changeFeed.
4. Observe that the streaming query starts but initialOffset() or latestOffset() never completes, no microbatch makes progress, and no failure is surfaced.

A deterministic unit-test reproduction is to return Mono.never() from the metadata request and verify that offset discovery currently blocks indefinitely.

Fix approach validated in our fork

We implemented and validated the following mitigation:

1. Introduce spark.cosmos.changeFeed.maxRetryDurationInSeconds, defaulting to 300 seconds and requiring a positive value.
2. Wrap change-feed initialOffset() and latestOffset() discovery in a shared monotonic deadline.
3. Apply that same deadline across nested metadata operations, retry attempts, and retry backoff; nested operations cannot extend the parent deadline.
4. Replace untimed metadata waits with Mono.block(remainingDuration), ensuring a stalled reactive subscription is canceled when the deadline expires.
5. Propagate the resulting failure to Spark so the streaming query fails and the orchestrator can retry.
6. Scope the deadline to change-feed offset metadata discovery, preserving existing catalog, ordinary read/write, partition-split, and microbatch-processing behavior.

We verified persistent transient failures, nested deadlines, cancellation of a genuinely noncompleting Mono.never() request, and interrupted-retry cleanup. Across 440 healthy production samples, offset discovery had a median of 0.41 seconds, a p99 of 46 seconds, and a maximum of 50 seconds, making the five-minute default conservative.

We can provide the implementation or a patch helpful.

[github-actions]

Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.

//cc: @kushagraThapar @pjohari-ms @TheovanKraay

[github-actions]

🎯 Agentic Issue Triage

Summary: The Azure Cosmos DB Spark connector (com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12) can block indefinitely during change-feed offset discovery (initialOffset()/latestOffset()) when a regional Cosmos endpoint cannot be resolved due to DNS failure, because internal Mono.block() calls are untimed. The author has validated a fix involving a configurable deadline (spark.cosmos.changeFeed.maxRetryDurationInSeconds) and is offering to contribute the implementation.

📋 Issue Details

Package: com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:4.49.2
Affected API: ChangeFeedMicroBatchStream.initialOffset() / latestOffset() — untimed Mono.block() in change-feed metadata/partition discovery
Scenarios:

• Cosmos private endpoint regional DNS record missing → UnknownHostException → connector hangs indefinitely
• Production incident: query stuck for ~2h 45m until external scheduler forced termination

Root ask: Introduce a configurable deadline for change-feed offset discovery so that persistent DNS/network failures cause a clean query failure rather than an indefinite hang, allowing Spark and orchestrators to retry.

🔎 Debugging / Reproduction Notes

The root cause is an untimed reactive Mono.block() inside offset discovery; DNS retries loop inside the reactive subscription and never surface to the connector's outer retry logic.

Suggested investigation steps:

1. Locate ChangeFeedMicroBatchStream.initialOffset() and latestOffset() in sdk/cosmos/azure-cosmos-spark_3-5_2-12 and identify all Mono.block() / Mono.block(Duration) call sites.
2. Confirm that no wall-clock deadline is imposed on the full metadata-discovery chain (feed-range enumeration + partition metadata).
3. Review the outer transient-error retry policy to understand how a stalled Mono.block() interacts with it (the outer loop cannot be entered while block() is hanging).
4. Evaluate the author's proposed fix: wrap offset discovery in a shared monotonic deadline propagated to nested operations, replacing untimed block() with block(remainingDuration).
5. Consider exposing spark.cosmos.changeFeed.maxRetryDurationInSeconds (default 300 s) as a new Spark config option and validate it does not affect catalog, read/write, or microbatch-processing paths.

Author has a validated fork implementation and production telemetry (p99 offset discovery: 46 s, max: 50 s over 440 samples) supporting a 300 s default.

🏷️ Label Confidence

• Category: Client — the issue is about SDK/connector behavior (untimed Mono.block()), not the REST API or Azure service itself
• Service: Cosmos — explicitly references com.azure.cosmos.spark, located in /sdk/cosmos/
• Confidence: High — artifact ID explicitly stated in the issue body; category and service are unambiguous; aligns with quality reference issue [QUERY] When will Azure Cosmos DB OLTP Spark 4 connector be ready? #46290 (Client + Cosmos + customer-reported + issue-addressed)

👥 Owner Routing

• Matched CODEOWNERS entry: # ServiceLabel: %Cosmos (line 295) — service label "Cosmos" matches directly
• AzureSdkOwners: none listed
• ServiceOwners: @kushagraThapar, @pjohari-ms, @TheovanKraay
• Routing action: Added "Service Attention" and "needs-team-attention" labels (customer-reported issue); issue left unassigned per ServiceOwners-only routing rules
• Scan notes: No multi-label Cosmos entry exists in CODEOWNERS; the single %Cosmos entry was the first and only match

Generated by Agentic Triage for #50021 · 29.7 AIC · ⌖ 6.63 AIC · ⊞ 11K · ◷