[SPARK-6376][SQL] Avoid eliminating subqueries until optimization

[marmbrus]

Previously it was okay to throw away subqueries after analysis, as we would never try to use that tree for resolution again. However, with eager analysis in DataFrames this can cause errors for queries such as:

val df = Seq(1,2,3).map(i => (i, i.toString)).toDF("int", "str")
df.as('x).join(df.as('y), $"x.str" === $"y.str").groupBy("x.str").count()

As a result, in this PR we defer the elimination of subqueries until the optimization phase.

[rxin]

nitpick here - can we put an explicit type?

[SparkQA]

Test build #29066 has finished for PR 5160 at commit 81cd597.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #29071 has finished for PR 5160 at commit 9137e03.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class TaskCommitDenied(jobID: Int, partitionID: Int, attemptID: Int) extends TaskFailedReason
  • class ExecutorSource(threadPool: ThreadPoolExecutor, executorId: String) extends Source
  • class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable
  • class NaiveBayesModel(Saveable, Loader):
  • class SqlParser extends AbstractSparkSQLParser with DataTypeParser
  • case class CombineSum(child: Expression) extends AggregateExpression
  • case class CombineSumFunction(expr: Expression, base: AggregateExpression)
  • protected[sql] class DataTypeException(message: String) extends Exception(message)

[yhuai]

Add a comment at here to let others know that the first step in Optimizer is to remove SubQueries (which are helper wrappers for query analysis)?

[marmbrus]

added

[SparkQA]

Test build #29100 has finished for PR 5160 at commit 27d25bf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

LGTM

[SparkQA]

Test build #29104 has finished for PR 5160 at commit a9bb262.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.