[SPARK-42452][BUILD] Remove hadoop-2 profile from Apache Spark 3.5.0#40788
[SPARK-42452][BUILD] Remove hadoop-2 profile from Apache Spark 3.5.0#40788LuciferYang wants to merge 3 commits into
hadoop-2 profile from Apache Spark 3.5.0#40788Conversation
LuciferYang
changed the title
hadoop-2 profilehadoop-2 profile from Apache Spark 3.5.0
|
Test first, need wait until 3.4 release at least |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you, @LuciferYang .
|
Also, cc FYI, @sunchao |
3.4.0 is uploaded to mirrors. |
I know, but let's wait for the official website update |
|
Personally I like this move, but not sure whether there are other users in the community that still depend on Hadoop 2.x. Has this been discussed before? |
yes, like this one Dropping Apache Spark Hadoop2 Binary Distribution? |
|
Oh cool, it's good then. Forgot about this thread even though I replied on it .. |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM (from my side)
|
Just sent an email to the community in case anyone still have concerns on this. |
|
friendly ping @dongjoon-hyun @sunchao |
|
Merged to master, thanks! |
|
Thanks @sunchao @dongjoon-hyun @bjornjorgensen ~ |
### What changes were proposed in this pull request? This PR aims to remove unused `commons-beanutils` dependency from `pom.xml` and `LICENSE-binary`. ### Why are the changes needed? #30701 removed `commons-beanutils` from `hadoop-3` profile at Apache Spark 3.2.0. - #30701 #40788 removed `hadoop-2` profile from Apache Spark 3.5.0 - #40788 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45705 from dongjoon-hyun/SPARK-47548. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR removes `Hadoop 3` from the GitHub Action workflow names. ### Why are the changes needed? Apache Spark 3.5+ supports only Hadoop 3, so `Hadoop 3` is redundant in the workflow names in these days because Apache Spark 3.4 and older are EOL. All live branches are `Hadoop 3`. - #40788 Note that this PR focuses only for GitHub Action workflow names, not a profile. We have kept the profile for the cases we need to split Hadoop distributions in a way. For example, hadoop-4 (although it doesn't exist yet) or hadoop-3.6 (if needed). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Closes #56576 from dongjoon-hyun/SPARK-57513. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
4 participants
What changes were proposed in this pull request?
This pr aims to remove
hadoop-2profile from Apache Spark 3.5.0.Why are the changes needed?
Spark 3.4.0 no longer releases Hadoop2 binary distribtuion(SPARK-42447) and Hadoop 2 GitHub Action job already removed after SPARK-42447, we can remove
hadoop-2profile from Apache Spark 3.5.0.Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions