Skip to content

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix)#1620

Closed
liancheng wants to merge 2 commits into
apache:masterfrom
liancheng:jdbc-with-maven-fix
Closed

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix)#1620
liancheng wants to merge 2 commits into
apache:masterfrom
liancheng:jdbc-with-maven-fix

Conversation

@liancheng

@liancheng liancheng commented Jul 28, 2014

Copy link
Copy Markdown
Contributor

JIRA issue: SPARK-2410

Another try for #1399 & #1600. Those two PR breaks Jenkins builds because we made a separate profile hive-thriftserver in sub-project assembly, but the hive-thriftserver module is defined outside the hive-thriftserver profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in hive-thriftserver, but tests fail because related .class files are not included in the assembly jar.

In the most recent commit, module hive-thriftserver is moved into its own profile to fix this problem. All previous commits are squashed for clarity.

liancheng added 2 commits July 29, 2014 01:27
@liancheng
Cherry picked the Hive Thrift server
ec3c7a7 
Starts Hive Thrift server via spark-submit

Make HiveThriftServer2 play well with spark-submit

Starts spark-sql shell with spark-submit

Updated docs for Hive compatibility and Shark migration guide draft

Fixed minor issues in spark-sql and start-thriftserver.sh

Added missing license headers

Fixed more license header issues

Adapts test suites to spark-submit settings

Addressed all comments by @pwendell

Updated documents and build scripts for the newly added hive-thriftserver profile

Starts beeline with spark-submit

Fixed spark-submit application options handling logic

Any options in the application option list with the same option name
that SparkSubmitArguments recognizes (e.g., --help) are stolen by
SparkSubmit instead of passed to the application.

Fixed failed test suites

Disabled MIMA for hive-thriftserver

Reordered spark-submit options in spark-shell[.cmd]

All options behind primary resource are (and should be) recognized as
application options now.

Updated Spark SQL programming guide docs

Revert changes related to SPARK-2678, decided to move them to another PR

Uses random port for HiveThriftServer2 to avoid collision with parallel builds
@liancheng
Moved hive-thriftserver module definition into its own profile
629988e
@SparkQA

SparkQA commented Jul 28, 2014

Copy link
Copy Markdown

QA tests have started for PR 1620. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@SparkQA

SparkQA commented Jul 28, 2014

Copy link
Copy Markdown

QA results for PR 1620:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17287/consoleFull

@marmbrus

marmbrus commented Jul 28, 2014

Copy link
Copy Markdown
Contributor

Hopefully third times a charm :)

Merged!

@liancheng

liancheng commented Jul 29, 2014

Copy link
Copy Markdown
Contributor Author

Hmm, didn't find related code in master, and this PR is still open, seems it's not successfully merged yet?

@marmbrus

marmbrus commented Jul 29, 2014

Copy link
Copy Markdown
Contributor

https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=a7a9d14479ea6421513a962ff0f45cb969368bab

GitHub is lagging... We filled a bug with infra.
On Jul 28, 2014 5:09 PM, "Cheng Lian" notifications@github.com wrote:

Hmm, didn't find related code in master, and this PR is still open, seems
it's not successfully merged yet?

—
Reply to this email directly or view it on GitHub
#1620 (comment).

@liancheng

liancheng commented Jul 29, 2014

Copy link
Copy Markdown
Contributor Author

Oh, really "lucky" these days :)

@asfgit asfgit closed this in a7a9d14 Jul 29, 2014
@liancheng liancheng mentioned this pull request Aug 2, 2014
Closed
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@liancheng @conviva-zz
[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile…
dc90bcc 
… fix)

JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410)

Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar.

In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits:

629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile
ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server
@liancheng liancheng deleted the jdbc-with-maven-fix branch September 24, 2014 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liancheng @SparkQA @marmbrus