[SQL]Extract the joinkeys from join condition by chenghao-intel · Pull Request #1190 · apache/spark

chenghao-intel · 2014-06-24T05:46:31Z

Extract the join keys from equality conditions, that can be evaluated using equi-join.

AmplabJenkins · 2014-06-24T05:50:13Z

Merged build triggered.

AmplabJenkins · 2014-06-24T05:50:19Z

Merged build started.

concretevitamin · 2014-06-24T05:53:38Z

nit: dot should precede its operator immediately

AmplabJenkins · 2014-06-24T07:26:56Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-24T07:26:56Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16049/

AmplabJenkins · 2014-06-24T07:45:15Z

Merged build triggered.

AmplabJenkins · 2014-06-24T07:45:25Z

Merged build started.

AmplabJenkins · 2014-06-24T09:20:17Z

Merged build finished.

AmplabJenkins · 2014-06-24T09:20:18Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16051/

chenghao-intel · 2014-06-25T04:26:17Z

Jenkins, retest this please.

chenghao-intel · 2014-06-25T04:29:42Z

@rxin , can you ask Jenkins to retest this? Seems he doesn't answer me. :)

AmplabJenkins · 2014-06-25T04:30:17Z

Merged build triggered.

AmplabJenkins · 2014-06-25T04:30:23Z

Merged build started.

chenghao-intel · 2014-06-25T04:31:42Z

Oh, Jenkins is working. :)

AmplabJenkins · 2014-06-25T06:07:06Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-25T06:07:06Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16104/

marmbrus · 2014-06-26T00:52:51Z

I'm not sure what the point of this change is. It is only serving to make the planner more brittle and tied to the specifics of the current implementation of the optimizer.

If the current pattern for hash joins is correct and more general, I think we should keep it.

chenghao-intel · 2014-06-26T01:05:13Z

The join/where predicate push down has been done in PushPredicateThroughJoin of the logical plan optimizer, I don't think we really need to do it again here. Hence I wrote an new pattern ExtractEquiJoinKeys to extract the join keys only, which should be more specific.

chenghao-intel · 2014-06-26T01:10:58Z

BTW, if I followed the current implementation pattern, which means I have to handle predicate push down for the outer join as it's done for inner join, too, that may make the code duplicated(with the optimizer) and confusing.

marmbrus · 2014-06-26T01:48:31Z

Okay, you've convinced me with the outer join argument. Remove HashFilteredJoin as its pretty redundant with your pattern.

marmbrus · 2014-06-26T01:48:41Z

and please rebase to master.

AmplabJenkins · 2014-06-26T03:15:20Z

Merged build triggered.

AmplabJenkins · 2014-06-26T03:15:26Z

Merged build started.

chenghao-intel · 2014-06-26T03:16:29Z

Thank you @marmbrus , updated, let's see the testing result.

AmplabJenkins · 2014-06-26T04:51:59Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-26T04:51:59Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16136/

egraldlo · 2014-06-26T08:42:11Z

maybe annotation on line 48 can be modified.

Good catch.

chenghao-intel · 2014-06-27T00:33:01Z

Thanks, updated.

AmplabJenkins · 2014-06-27T00:35:24Z

Merged build triggered.

AmplabJenkins · 2014-06-27T00:35:29Z

Merged build started.

AmplabJenkins · 2014-06-27T02:12:53Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-27T02:12:53Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16177/

marmbrus · 2014-06-27T02:18:34Z

Thanks, merged into master.

Extract the join keys from equality conditions, that can be evaluated using equi-join. Author: Cheng Hao <hao.cheng@intel.com> Closes apache#1190 from chenghao-intel/extract_join_keys and squashes the following commits: 4a1060a [Cheng Hao] Fix some of the small issues ceb4924 [Cheng Hao] Remove the redundant pattern of join keys extraction cec34e8 [Cheng Hao] Update the code style issues dcc4584 [Cheng Hao] Extract the joinkeys from join condition

chenghao-intel changed the title ~~Extract the joinkeys from join condition~~ [SQL]Extract the joinkeys from join condition Jun 24, 2014

concretevitamin reviewed Jun 24, 2014
View reviewed changes

chenghao-intel mentioned this pull request Jun 25, 2014

[SQL][SPARK-2212]Hash Outer Join #1147

Closed

chenghao-intel added 3 commits June 26, 2014 10:03

Extract the joinkeys from join condition

dcc4584

Update the code style issues

cec34e8

Remove the redundant pattern of join keys extraction

ceb4924

egraldlo reviewed Jun 26, 2014
View reviewed changes

Fix some of the small issues

4a1060a

asfgit closed this in 981bde9 Jun 27, 2014

chenghao-intel deleted the extract_join_keys branch June 27, 2014 05:04

Uh oh!

Conversation

chenghao-intel commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

concretevitamin Jun 24, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

AmplabJenkins commented Jun 24, 2014

Uh oh!

chenghao-intel commented Jun 25, 2014

Uh oh!

chenghao-intel commented Jun 25, 2014

Uh oh!

AmplabJenkins commented Jun 25, 2014

Uh oh!

AmplabJenkins commented Jun 25, 2014

Uh oh!

chenghao-intel commented Jun 25, 2014

Uh oh!

AmplabJenkins commented Jun 25, 2014

Uh oh!

AmplabJenkins commented Jun 25, 2014

Uh oh!

marmbrus commented Jun 26, 2014

Uh oh!

chenghao-intel commented Jun 26, 2014

Uh oh!

chenghao-intel commented Jun 26, 2014

Uh oh!

marmbrus commented Jun 26, 2014

Uh oh!

marmbrus commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

chenghao-intel commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

egraldlo Jun 26, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus Jun 26, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Jun 27, 2014

Uh oh!

AmplabJenkins commented Jun 27, 2014

Uh oh!

AmplabJenkins commented Jun 27, 2014

Uh oh!

AmplabJenkins commented Jun 27, 2014

Uh oh!

AmplabJenkins commented Jun 27, 2014

Uh oh!

marmbrus commented Jun 27, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone