Skip to content

[BEAM-8740] Remove unused dependency from Spark runner #10564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lukecwik merged 2 commits into from
Jan 13, 2020
Merged

[BEAM-8740] Remove unused dependency from Spark runner #10564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
302bb9b
[BEAM-8740] Remove unused dependency from Spark runner
iemejia Jan 13, 2020
04a79fb
Add UsesTestStream category on ParDoTest$TimerTests.testOutputTimestamp
iemejia Jan 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,6 @@ class BeamModulePlugin implements Plugin<Project> {
def protobuf_version = "3.11.1"
def quickcheck_version = "0.8"
def spark_version = "2.4.4"
def spark_structured_streaming_version = "2.4.0"

// A map of maps containing common libraries used per language. To use:
// dependencies {
Expand Down
1 change: 0 additions & 1 deletion runners/spark/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ dependencies {
provided "com.esotericsoftware.kryo:kryo:2.21"
runtimeOnly library.java.jackson_module_scala
runtimeOnly "org.scala-lang:scala-library:2.11.8"
compile "org.scala-lang.modules:scala-java8-compat_2.11:0.9.0"
testCompile project(":sdks:java:io:kafka")
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
// SparkStateInternalsTest extends abstract StateInternalsTest
Expand Down
1 change: 1 addition & 0 deletions sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3736,6 +3736,7 @@ public void testTwoTimersSettingEachOtherWithCreateAsInput() {
ValidatesRunner.class,
UsesStatefulParDo.class,
UsesTimersInParDo.class,
UsesTestStream.class,
Copy link
Copy Markdown
Member Author

@iemejia iemejia Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add this one because it was not correctly categorized and VR were breaking

Copy link
Copy Markdown
Member Author

@iemejia iemejia Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems is reported here
https://issues.apache.org/jira/browse/BEAM-9083

Copy link
Copy Markdown
Member

@lukecwik lukecwik Jan 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the VR tests exclude UsesTestStreamWithOutputTimestamp?

Copy link
Copy Markdown
Member Author

@iemejia iemejia Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the line exactly below that one. Notice that Spark runner does not implement TestStream so we probably need to make the test belong to both categories.

Copy link
Copy Markdown
Member Author

@iemejia iemejia Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind to maybe lgtm / merge this one @lukecwik. Seems most VR are broken because of it.

Copy link
Copy Markdown
Member

@lukecwik lukecwik Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I was under the impression that we would place things in the most appropriate category but now thinking about it makes sense to place it in this category as well.

UsesTestStreamWithOutputTimestamp.class
})
public void testOutputTimestamp() {
Expand Down