Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
18ce1b8
Add capability to inherit SessionState (SQL conf, temp tables, regist…
kunalkhamar Feb 3, 2017
9beb78d
Add tests for forking new session with inherit config enabled. Update…
kunalkhamar Feb 6, 2017
a343d8a
Fix constructor default args for bytecode compatibility.
kunalkhamar Feb 6, 2017
4210079
Incorporate feedback. Fix association of incorrect SparkSession while…
kunalkhamar Feb 10, 2017
6da6bda
Update spark version. Rename clone to copy, in order to avoid Java Ob…
kunalkhamar Feb 10, 2017
579d0b7
Make lazy vals strict.
kunalkhamar Feb 14, 2017
2837e73
Refactor SessionState to remove passing of base SessionState, and ini…
kunalkhamar Feb 16, 2017
8c00344
Remove unused import.
kunalkhamar Feb 16, 2017
f423f74
Remove SparkSession reference from SessionState.
kunalkhamar Feb 16, 2017
b1371d8
Merge branch 'master' into fork-sparksession
kunalkhamar Feb 16, 2017
2cee190
Fix initialization loop.
kunalkhamar Feb 17, 2017
e2bbfa8
Fix var name error.
kunalkhamar Feb 17, 2017
8ac778a
Add tests. Refactor. Temporarily disable subtest SPARK-18360: default…
kunalkhamar Feb 18, 2017
0c732ce
Merge branch 'master' into fork-sparksession
kunalkhamar Feb 21, 2017
3c995e1
Fix copy of SessionCatalog. Changes from review.
kunalkhamar Feb 21, 2017
292011a
Merge branch 'fork-sparksession' of github.com:kunalkhamar/spark into…
kunalkhamar Feb 21, 2017
b027412
Merge branch 'master' into fork-sparksession
kunalkhamar Feb 21, 2017
295ee41
Add synchronized blocks. Ignore hive metastore tests for now.
kunalkhamar Feb 21, 2017
847b484
Merge branch 'fork-sparksession' of github.com:kunalkhamar/spark into…
kunalkhamar Feb 21, 2017
9beba84
Add tests. Force copy of session state on cloneSession.
kunalkhamar Feb 22, 2017
3d2e4a6
Rename copy to clone() to work around copy method of case classes. Mo…
kunalkhamar Feb 22, 2017
4f70d12
Fix HiveSessionState clone.
kunalkhamar Feb 22, 2017
dd2dedd
Add tests for HiveSessionState. Review feedback.
kunalkhamar Feb 23, 2017
8a8d47b
Simplify TestSQLContext. Review feedback.
kunalkhamar Feb 23, 2017
ffc2058
(attempt to) Fix tests.
kunalkhamar Feb 24, 2017
16824f9
Review.
kunalkhamar Feb 24, 2017
fd11ee2
Update test case.
kunalkhamar Feb 24, 2017
437b0bc
Add throwing exception if wrong SessionState clone is called. Update …
kunalkhamar Feb 25, 2017
300d3a0
Most of the changes from review.
kunalkhamar Feb 28, 2017
3ee271f
All but one review feedback.
kunalkhamar Mar 6, 2017
c3f052f
Merge remote-tracking branch 'origin/master' into pr16826
zsxwing Mar 6, 2017
0bdc81c
Merge remote-tracking branch 'origin/master' into pr16826
zsxwing Mar 6, 2017
2740c63
Fix tests
zsxwing Mar 6, 2017
0f167db
Clean up tests
zsxwing Mar 6, 2017
2f0b1ad
fix SessionCatalogSuite
zsxwing Mar 7, 2017
c41e7bc
More cleanup
zsxwing Mar 7, 2017
5eb6733
More tests
zsxwing Mar 7, 2017
05abcf8
Update tests and a param comment.
kunalkhamar Mar 7, 2017
4c23e7a
Merge branch 'master' into fork-sparksession
kunalkhamar Mar 8, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Review.
  • Loading branch information
kunalkhamar committed Feb 24, 2017
commit 16824f916e87fd90706f9dfd7b7dd81d87b732dd
Original file line number Diff line number Diff line change
Expand Up @@ -1004,7 +1004,7 @@ object SparkSession {

/**
* Helper method to create an instance of `SessionState` based on `className` from conf.
* The result is either `SessionState` or `HiveSessionState`
* The result is either `SessionState` or `HiveSessionState`.
*/
private def instantiateSessionState(
className: String,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,7 @@ private[hive] class HiveClientImpl(
val original = Thread.currentThread().getContextClassLoader
// Set the thread local metastore client to the client associated with this HiveClientImpl.
Hive.set(client)
// replace conf in cached hive with current conf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A get after a set looks weird. Actually I need to go back to check Hive source to know get will replace conf with provided one. Shall we move this into private def client: Hive and add comment for it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@viirya Hive.get(conf) replaces conf in the thread local Hive with current conf, so it must be called after Hive.set(client) which sets the current thread local Hive.

Hive.get(conf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because IsolatedClientLoader is shared? If we do not make this change, any test case failed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of reusingIsolatedClientLoader.cachedHive. Without this line, it may use an out-of-date HiveConf. The failed test is HiveSparkSubmitSuite.test("SPARK-18360: default table path of tables in default database should depend on the " + "location of default database")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

// The classloader in clientLoader could be changed after addJar, always use the latest
// classloader
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class HiveSessionCatalogSuite extends SessionCatalogSuite {
new Configuration(),
new SimpleFunctionRegistry,
CatalystSqlParser)
assert(original ne clone)
assert(clone.getTempView("copytest1") == Option(tempTable1))

// check if clone and original independent
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,8 @@ import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql._
import org.apache.spark.sql.hive.test.TestHiveSingleton

class HiveSessionStateSuite
extends SessionStateSuite
with TestHiveSingleton
with BeforeAndAfterEach {

class HiveSessionStateSuite extends SessionStateSuite
with TestHiveSingleton with BeforeAndAfterEach {

override def beforeEach(): Unit = {
createSession()
Expand Down