[SPARK-10304][SQL] Partition discovery should throw an exception if the dir structure is invalid

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-10304

This patch detects if the structure of partition directories is not valid.

The test cases are from #8547. Thanks @zhzhan.

cc @liancheng

[SparkQA]

Test build #42708 has finished for PR 8840 at commit 2d13156.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #42718 has finished for PR 8840 at commit 2d13156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[JoshRosen]

@chenghao-intel, it looks like this PR's error message improvements are similar to the ones that you added as part of #8026. @viirya, could you take a look at @chenghao-intel's PR to see which approach you like better?

[SparkQA]

Test build #44001 has finished for PR 8840 at commit 2d13156.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@JoshRosen ok, I will take a look at it.

[chenghao-intel]

Move this assert out of the resolvePartitions, since it's not so tight with the original purpose of this function, and we can avoid the code change for this function.

[SparkQA]

Test build #44295 has finished for PR 8840 at commit 779fbd2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44316 has finished for PR 8840 at commit cdf6dc4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@chenghao-intel @JoshRosen any comments? Is this patch ready?

[davies]

Can we writhe these like this?

val (basePaths, pathsWithPartitionValues) = paths.flatMap {
}.unzip

[davies]

LGTM

[SparkQA]

Test build #44888 has finished for PR 8840 at commit 3db4226.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #44906 has finished for PR 8840 at commit 3db4226.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Should this one be backported to branch-1.5?

cc @liancheng @yhuai

[yhuai]

Can you give a case that we will hit this branch? What will basePaths be at here?

[yhuai]

Which case is for the one I mentioned in the jira?

[yhuai]

@rxin I think it is fine to not backport it. Without this, a user will get the DataFrame and when he/she queries it, it will fail (and the error message does not say what's the cause). Since, it does not really let users make any real progress, I think we do not really need to backport it.

[rxin]

OK great. @viirya can you submit a followup pr to address @yhuai's feedback.

[viirya]

@rxin @yhuai OK. Submitted a following pr at #9459.

[yhuai]

Why do we need !columns.isEmpty?

[yhuai]

oh i see. It is for something like table/a=1/_temporary/something, right?