[SQL] SPARK-6981: Factor out SparkPlanner and QueryExecution from SQLContext

[evacchi]

Cleaned-up version of PR #5556

[smola]

Shouldn't this be private[sql] now?

[evacchi]

It is protected[sql] in master.

BTW, this PR is geared towards making it easier for third parties (and HiveContext) to add new processing rules without requiring to subclass (see PR #5556 and SPARK-6320)

I would actually advise making these classes public (or at least protected, without the [sql] qualifier)

[evacchi]

If everybody agrees, I think we can restart the Jenkins build (this is just the same as the other PR, after all)

[evacchi]

@rxin may I ask if you can trigger a test build? code is the same as PR #5556

[rxin]

Jenkins, test this please.

[SparkQA]

Test build #33237 has finished for PR 6122 at commit ac03efe.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait QueryPlanner[PhysicalPlan <: TreeNode[PhysicalPlan]]
  • protected[sql] class QueryExecution(val sqlContext: SQLContext, val logical: LogicalPlan)
  • protected[sql] class SparkPlanner(val sqlContext: SQLContext) extends SparkStrategies
  • protected[sql] class HiveQueryExecution(hiveContext: HiveContext, logicalPlan: LogicalPlan)

[evacchi]

Build fails because of this:

[info] spark-sql: found 3 potential binary incompatibilities (filtered 328)
[error]  * class org.apache.spark.sql.SQLContext#SparkPlanner does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.SQLContext$SparkPlanner")
[error]  * method prepareForExecution()org.apache.spark.sql.catalyst.rules.RuleExecutor in class org.apache.spark.sql.SQLContext does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.sql.SQLContext.prepareForExecution")
[error]  * class org.apache.spark.sql.SQLContext#QueryExecution does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.SQLContext$QueryExecution")

this is obvious, since that's the purpose of the PR. How should we proceed?

[evacchi]

Possible solution: add shims to preserve binary compatibility as follows:

class SQLContext {
    ...
   @deprecated
    class SparkPlanner extends org.apache.spark.sql.SparkPlanner
   @deprecated
    class QueryExecution extends org.apache.spark.sql.QueryExecution 
   @deprecated
    lazy val prepareForExecution = ...
    ...
}

class QueryExecution(sqlContext: SQLContext) {
   ...
   lazy val prepareForExecution = sqlContext.prepareForExecution
   ...
}

etc.

However, I wouldn't do that unless it is really necessary, because it really makes extending the non-deprecated classes a bit awkward (must be careful with imports)

[evacchi]

addressed in PR #6356

[AmplabJenkins]

Can one of the admins verify this patch?