[SPARK-26576][SQL] Broadcast hint not applied to partitioned table#23507
[SPARK-26576][SQL] Broadcast hint not applied to partitioned table#23507jzhuge wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
can we check the physical plan to make sure BroadcastJoin is picked?
There was a problem hiding this comment.
+1. And since master doesn't use ResolvedHint in optimization anymore, this issue doesn't exist in master, but figured it wouldn't hurt to add a test in master too.
There was a problem hiding this comment.
Fine with porting at least the unit test to master.
|
ok to test |
|
Test build #101013 has finished for PR 23507 at commit
|
|
Test build #101035 has finished for PR 23507 at commit
|
There was a problem hiding this comment.
The bug is not here. We should get rid of this lines:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala#L71-L72
There was a problem hiding this comment.
I have tried this fix, it works. However, I am not sure about the original rational to add this code in SPARK-14581, and trying to minimize the impact of removing it because there are other callers of PhysicalOperation.unapply.
If @davies and @cloud-fan are ok and we can pass test cases covering the scenarios for this code, I'd be happy to go with this fix.
There was a problem hiding this comment.
seems better to remove that line.
|
Test build #101035 failures seem unrelated. VersionSuite and HiveClientSuites pass locally for me. |
|
Test build #101053 has finished for PR 23507 at commit
|
|
retest this please |
|
Test build #101082 has finished for PR 23507 at commit
|
|
LGTM. Thanks! Merged to 2.4 and 2.3. |
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes #23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes #23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com> (cherry picked from commit b9eb0e8) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
|
Thx @gatorsmile! Should we merge it to master any way? Remove the buggy/dead code and add a unit test. |
|
The buggy code doesn't exist in master anymore, but it will be great to put the test in master. Can you send a PR to do it? thanks! |
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite Closes apache#23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com> (cherry picked from commit b9eb0e8)
|
I'm not 100% sure, but I think this is causing StreamingQuerySuite to fail consistently in 2.3, but not 2.4. See failures from https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ and drill down to things like https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.6/577/testReport/ . Most of the subsequent build failures seem to be related to this test. CC @maropu as this probably affects a 2.3.3 release. |
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite Closes #23507 from jzhuge/SPARK-26576. Closes #23530 from jzhuge/SPARK-26576-master. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
|
Sorry, but the rightest column in the page @gatorsmile showed means what? It seems |
|
I tried to run the unit tests |
|
It's possible it merely uncovered a different problem, in the test or main code, that only exists in 2.3. How important is the change for 2.3? given that there is a release imminent, would it be OK to revert it now in 2.3, or does that lose an important fix? |
|
+1 for reverting it from v2.3 (it might be worth digging into the cause though, IMO its not worth blocking the v2.3 release). |
|
I am fine to revert it, although I do not think they are related.
…On Mon, Jan 14, 2019 at 5:15 PM Takeshi Yamamuro ***@***.***> wrote:
+1 for reverting it from v2.3 (it might be worth digging into the cause
though, IMO its not worth blocking the v2.3 release).
If other guys agree to that, I'll do it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#23507 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ALCApfzW_cmV9X40QWQUEhF2gwSbOH3Rks5vDSu0gaJpZM4Z4-eK>
.
|
|
+1 on reverting
Failed to reproduce the issue. My internal Jenkins job mimic'ng
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/572/
completed
16,477 tests successfully.
14:32:59 + build/mvn -version
14:33:01 Using `mvn` from path:
/mnt/builds/slave/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/mvn
14:33:01 Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T16:41:47+00:00)
14:33:01 Maven home:
/mnt/builds/slave/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9
14:33:01 Java version: 1.8.0_192, vendor: Azul Systems, Inc.
14:33:01 Java home: /usr/lib/jvm/zulu-8-amd64/jre
14:33:01 Default locale: en_US, platform encoding: UTF-8
14:33:01 OS name: "linux", version: "4.4.0-1072-aws", arch: "amd64",
family: "unix"
14:33:01 + build/mvn clean package --batch-mode -DskipTests -Phadoop-2.7
-Pyarn -Phive -Phive-thriftserver -Pkinesis-asl -Pmesos
14:43:04 Executing Maven: -B -f
/mnt/builds/slave/workspace/jzhuge-test2/pom.xml
-Dmaven.repo.local=/mnt/builds/slave/workspace/jzhuge-test2/.repository
test -DzincPort=6666 --fail-at-end -Phadoop-2.7 -Pyarn -Phive
-Phive-thriftserver -Pkinesis-asl -Pmesos
…On Mon, Jan 14, 2019 at 6:50 AM Sean Owen ***@***.***> wrote:
It's possible it merely uncovered a different problem, in the test or main
code, that only exists in 2.3.
How important is the change for 2.3? given that there is a release
imminent, would it be OK to revert it now in 2.3, or does that lose an
important fix?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#23507 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABy-pAvV0YizQrVOUP-I-RHzXaKeVAo8ks5vDJk_gaJpZM4Z4-eK>
.
--
John Zhuge
|
|
What I really concern about is; should we fix the |
|
That would be ideal, of course; I have no idea what the issue is or difference is with 2.4. Up to you whether you want to investigate and hold up 2.3.3, or just punt on it and revert this commit and go ahead. |
|
ok, for now, I'll revert the commit and start the vote for the v2.3.3 release. |
|
I reverted this commit from branch-2.3, and then added a tag for v2.3.3-rc1 in the head. I'm re-publishing a candidate now, so I'll start a first vote for v2.3.3 in a few days after the Jenkins tests checked. |
|
Thanks! @maropu |
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite Closes apache#23507 from jzhuge/SPARK-26576. Closes apache#23530 from jzhuge/SPARK-26576-master. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes apache#23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com> (cherry picked from commit b9eb0e8) Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala
Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes apache#23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com> (cherry picked from commit b9eb0e8) Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes apache#23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
## What changes were proposed in this pull request? Make sure broadcast hint is applied to partitioned tables. Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4. ## How was this patch tested? - A new unit test in PruneFileSourcePartitionsSuite - Unit test suites touched by SPARK-14581: JoinOptimizationSuite, FilterPushdownSuite, ColumnPruningSuite, and PruneFiltersSuite cloud-fan davies rxin Closes apache#23507 from jzhuge/SPARK-26576. Authored-by: John Zhuge <jzhuge@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
7 participants
What changes were proposed in this pull request?
Make sure broadcast hint is applied to partitioned tables.
Since the issue exists in branch 2.0 to 2.4, but not in master, I created this PR for branch-2.4.
How was this patch tested?
@cloud-fan @davies @rxin