Skip to content

[SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-8489 test for Scala 2.12#22308

Closed
da-liii wants to merge 1 commit into
apache:masterfrom
da-liii:SPARK-8489-FOLLOWUP
Closed

[SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-8489 test for Scala 2.12#22308
da-liii wants to merge 1 commit into
apache:masterfrom
da-liii:SPARK-8489-FOLLOWUP

Conversation

@da-liii

@da-liii da-liii commented Sep 1, 2018

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

remove test-2.10.jar and add test-2.12.jar.

How was this patch tested?

$ sbt -Dscala-2.12
> ++ 2.12.6
> project hive
> testOnly *HiveSparkSubmitSuite -- -z "8489"

@SparkQA

SparkQA commented Sep 1, 2018

Copy link
Copy Markdown

Test build #95576 has finished for PR 22308 at commit 221fcac.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

srowen commented Sep 1, 2018

Copy link
Copy Markdown
Member

Oh what's the change here in the .jar file? just copied the jar file for 2.11/2.10?

@da-liii

da-liii commented Sep 2, 2018

Copy link
Copy Markdown
Contributor Author

Remove the deprecated 2.10 jar.

And build a 2.12 jar.

@da-liii

da-liii commented Sep 2, 2018

Copy link
Copy Markdown
Contributor Author

see #12924 and #11744

@da-liii

da-liii commented Sep 2, 2018

Copy link
Copy Markdown
Contributor Author

@felixcheung felixcheung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we be careful about having jar in the source tree? (that goes into a release?)

@srowen

srowen commented Sep 2, 2018

Copy link
Copy Markdown
Member

@sadhen I mean is this just a copy of the 2.10 jar? what's in the jar?
@felixcheung in this case there's a test specifically testing loading of a jar file, so it has to have a jar file to load. This code isn't part of the non-test code. (The source is there too FWIW)

@da-liii

da-liii commented Sep 3, 2018

Copy link
Copy Markdown
Contributor Author

@srowen The 2.12 jar is compiled and packaged from Main.scala and MyCoolClass.scala. Not a copy of 2.10 jar. Diff it, you will verify it.

The steps to generate it:

mvn install // install the 2.4.0-SNAPSHOT at project root
sbt package // package at sql/hive/src/test/resources/regression-test-SPARK-8489

This is the build.sbt:

scalaVersion := 2.12.6
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % "2.4.0-SNAPSHOT"
)

@srowen srowen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that's fine. That makes sense, and it's what the new JAR should be.

@srowen

srowen commented Sep 3, 2018

Copy link
Copy Markdown
Member

Merged to master

@asfgit asfgit closed this in 64bbd13 Sep 3, 2018
LuciferYang added a commit that referenced this pull request Sep 26, 2023
…ry files

### What changes were proposed in this pull request?
The purpose of this pr is to clean up the binary files used to assist with Scala 2.12 testing.

They include:
- `core/src/test/resources/TestHelloV3_2.12.jar` and `core/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44246(#41789).
- `connector/connect/client/jvm/src/test/resources/udf2.12` and `connector/connect/client/jvm/src/test/resources/udf2.12.jar` added by SPARK-43744(#42069)
- `connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44293(#41844)
- `sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar` added by SPARK-25304(#22308)

### Why are the changes needed?
Spark 4.0 no longer supports Scala 2.12.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #43106 from LuciferYang/SPARK-45321.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants