[FEATURE REQ][Spark Connector]Add spark 4.1 support

[xinlian12]

Summary

Add Spark 4.1 support via a new azure-cosmos-spark_4-1_2-13 module. Addresses SPARK-52787 package reorganization where HDFSMetadataLog moved from o.a.s.sql.execution.streaming to o.a.s.sql.execution.streaming.checkpointing.

Design

Create a shared azure-cosmos-spark_4 base module (similar to azure-cosmos-spark_3 for 3.x), then have both azure-cosmos-spark_4-0_2-13 and azure-cosmos-spark_4-1_2-13 inherit from it.

Why a shared base module?

Analysis shows all 18 Spark 4.x override files (12 main + 6 test Scala files) are identical between 4-0 and 4-1. Only 3 files differ (ChangeFeedInitialOffsetWriter.scala, CosmosCatalogBase.scala, CosmosCatalogITestBase.scala) due to the HDFSMetadataLog import change.

Steps

1. Create azure-cosmos-spark_4 — shared base module containing the 12 identical main Scala files + 6 identical test files that override azure-cosmos-spark_3 for Spark 4.x API compatibility
2. Refactor azure-cosmos-spark_4-0_2-13 — change parent to azure-cosmos-spark_4, remove duplicated files, keep only 4-0 specific config
3. Create azure-cosmos-spark_4-1_2-13 — inherits from azure-cosmos-spark_4, overrides the 3 files with updated HDFSMetadataLog imports, uses Spark 4.1.0 dependency
4. Update CI/pipeline configs — emulator matrices, spark.yml, aggregate-reports, version entries

Reference

• PR Add Spark 4.1 connector module (azure-cosmos-spark_4-1_2-13) #48688 (community contribution, similar but without shared module)
• Existing pattern: azure-cosmos-spark_3 → azure-cosmos-spark_3-x_2-12 / azure-cosmos-spark_3-x_2-13

Key difference from MetadataVersionUtil

With latest code on upstream-main, the spark connector has already removed the dependency on MetadataVersionUtil (inlined the version validation logic). So only HDFSMetadataLog import needs updating for Spark 4.1.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

[xinlian12]

Updated Design Decision

Analysis shows all 18 Spark 4.x override files (12 main + 6 test) are identical between azure-cosmos-spark_4-0_2-13 and azure-cosmos-spark_4-1_2-13. Only 3 files differ (HDFSMetadataLog import change).

New approach: Create a shared azure-cosmos-spark_4 base module (like azure-cosmos-spark_3), then have both 4-0 and 4-1 inherit from it. This avoids duplicating 18 files.

Closing PR #48860 in favor of this redesign.

[xinlian12]

Additional Task: Add Spark 4.1 Databricks Live Test Pipeline

Add a Databricks live test stage for Spark 4.1 in sdk/cosmos/spark.yml.

Details

• Cluster name: oltp-ci-spark40-2workers-d4dsv5-18.1
• Cluster URL: https://adb-1728595891330011.11.azuredatabricks.net/compute/clusters/0420-154401-1u8x3mb7?o=1728595891330011
• SparkVersion: azure-cosmos-spark_4-1_2-13
• JarName: azure-cosmos-spark_4-1_2-13-latest-ci-candidate.jar
• Use next sequential container/DB: sampleContainer9/sampleDB9
• JarReadOnlySasUriIndex: 3
• AvoidDBFS: true
• JavaVersion: 1.21

Follow the existing Spark 4.0 entry pattern in spark.yml.