[FEATURE REQ] Contribution for Spark 4.0 OLTP Connector for Cosmos DB

[pramodpatil07]

I am planning to upgrade our production data landscape (Databricks) to DBR 17.3 LTS, which runs on Apache Spark 4.0. Currently, the azure-cosmos-spark connector only supports up to Spark 3.5.

I am aware of issue #46290 stating a planned release for Q1 2026. However, due to our internal upgrade timelines, we need this support sooner.

Proposal I am a Senior Data Engineer with extensive Scala/Spark experience and I am willing to contribute the development of the Spark 4.0 module to this repository.

My proposed plan:

Create a new module sdk/cosmos/azure-cosmos-spark_4-0_2-13.

Port the existing 3.5 logic to support Scala 2.13 (required by Spark 4.0).

Address breaking changes in the Spark 4.0 DataSource V2 API.

Question for Maintainers (@kushagraThapar / @TheovanKraay)

Has internal work already started on this?

If yes, is there a feature branch I can contribute to?

If no, are you open to accepting a PR for this new module if I follow the contribution guidelines?

Setup

OS: Windows 11

IDE: IntelliJ / VS Code

Build: Maven 3.8+ / Java 17

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

[kushagraThapar]

@pramodpatil07 yes, we would like the contributions very much. I don't think any internal work has been started on this yet, however, I will defer to @TheovanKraay and @FabianMeiswinkel to answer that question.
We will get back to you shortly, thanks for your interest in providing support for Spark 4, we appreciate it

[FabianMeiswinkel]

@pramodpatil07 - as @kushagraThapar mentioned we are not only willing but glad to accept contributions in general. For the Spark 4 support I want to be honest - I think it is probably not a good use of your time. This task is not primarily a development task - the main challenges are around adding test support (live tests can only run in MSFT internal DevOps pipelines - and modifiying the Spark livetests to add Spark 4 coverage as an external contributor will be slow and painful (each change would require an MSFT employee to trigger pipeline runs to iterate on the changes). Overall the work stream to add Spark 4 support has two steps 1) Add scala 2.13 based module for Spark 3.5 (this work I have started already - see #47492) then 2) Adding the Spark 4 module - the latter will be relatively easy after we have added scala 2.13 support to test/build infrastructure in step 1).