Add Spark 4.1 connector module (azure-cosmos-spark_4-1_2-13) - #48688
Add Spark 4.1 connector module (azure-cosmos-spark_4-1_2-13)#48688mattslack-db wants to merge 2 commits into
Conversation
Add support for Apache Spark 4.1 which reorganized internal streaming packages (SPARK-52787). The following classes moved from `o.a.s.sql.execution.streaming` to `o.a.s.sql.execution.streaming.checkpointing`: - HDFSMetadataLog - MetadataVersionUtil This module overrides ChangeFeedInitialOffsetWriter, CosmosCatalogBase, and CosmosCatalogITestBase with updated imports, and uses a source-copy approach with excludes to avoid duplicate class definitions from the shared azure-cosmos-spark_3 source. Co-authored-by: Isaac
github-actions
Bot
added
Community Contribution
|
Thank you for your contribution mattslack-db! We will review the pull request and get back to you soon. |
|
@microsoft-github-policy-service agree [company="Databricks"] |
|
@microsoft-github-policy-service agree company="Databricks" |
|
mattslack-db - thanks for looking into this. To enable Spark 4.1 there are also several live-tests in the CI pipeline that would need to be enabled - and for security reasons starting those is only possible for devs (MSFT FTEs) with certain permisisons - which makes it really hard for someone external to contribute those. I think the best approach is for us to pick-up the changes, add the necessary live tests and then release Spark 4.1. Usually we try to do so a few weeks after Databricks releases a LTS version for the Spark runtime. Is there a specific need/urgency why you need Spark 4.1 support sooner? If so, please feel to reach out (also privately at fabianm at microsoft.com if needed) and we can work out a timeline? |
|
Sure no worries. I have a customer who wanted to use it with DBR18, and there is a breaking change in the API (see above - MetadataVersionUtil). That's the main reason for the PR. |
|
Ran into the issue that this resolves today. thanks mattslack-db for the PR |
|
Fixed in #48861 |
|
Great stuff, thanks |
3 participants
Summary
azure-cosmos-spark_4-1_2-13module for Apache Spark 4.1 supportHDFSMetadataLogandMetadataVersionUtilmoved fromo.a.s.sql.execution.streamingtoo.a.s.sql.execution.streaming.checkpointingazure-cosmos-spark_4-0_2-13module with updated imports for relocated classesChanges
ChangeFeedInitialOffsetWriter.scala— updated imports forHDFSMetadataLogandMetadataVersionUtilCosmosCatalogBase.scala— updated import forHDFSMetadataLogCosmosCatalogITestBase.scala(test) — updated import forHDFSMetadataLogpom.xml— Spark 4.1.0 dependency, source-copy approach with excludes for overridden filesTest plan
This pull request was AI-assisted by Isaac.