[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2 by panbingkun · Pull Request #37588 · apache/spark

panbingkun · 2022-08-20T11:41:41Z

What changes were proposed in this pull request?

The pr aim to implement v2 SHOW TABLE EXTENDED as ShowTableExec

Why are the changes needed?

To have feature parity with the datasource V1.

Does this PR introduce any user-facing change?

Yes, Support SHOW TABLE EXTENDED in v2.

How was this patch tested?

Add new UT.
By running the unified tests for v2 implementation:

$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowTablesSuite"
$ build/sbt "test:testOnly *ShowTablesSuite"

panbingkun · 2022-08-20T22:52:34Z

cc @MaxGekk

AmplabJenkins · 2022-08-21T07:27:40Z

Can one of the admins verify this patch?

Fokko · 2022-11-28T14:12:05Z

@panbingkun @MaxGekk Any progress on this? Thanks!

ja-michel · 2022-11-30T08:43:14Z

+1

rshanmugam1 · 2022-12-02T23:42:56Z

+1

MaxGekk · 2023-02-01T07:06:56Z

@panbingkun Could you resolve conflicts when you have time, please.

panbingkun · 2023-02-05T01:38:55Z

@panbingkun Could you resolve conflicts when you have time, please.

Done

xkrogen · 2023-02-13T16:23:02Z

@MaxGekk are you able to take another look at this PR? Would love to see it go in, this is an annoying feature gap that we need to work around currently for DSv2 sources :(

MaxGekk · 2023-02-13T16:31:13Z

are you able to take another look at this PR?

No, I haven't looked at this yet because of my wedding. :-)

this is an annoying feature gap that we need to work around currently for DSv2 sources :(

@xkrogen If you really need this feature, please, review this PR. I will join to you slightly later.

MaxGekk · 2023-02-16T09:22:42Z

-    pattern: Option[String]) extends V2CommandExec with LeafExecNode {
+    pattern: Option[String],
+    isExtended: Boolean = false,
+    partitionSpec: Option[TablePartitionSpec] = None) extends V2CommandExec with LeafExecNode {


TablePartitionSpec is legacy one, can't you use ResolvedPartitionSpec? For example, see

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowPartitionsExec.scala

Line 36 in 0494dc9

partitionSpec: Option[ResolvedPartitionSpec]) extends V2CommandExec with LeafExecNode {

MaxGekk · 2023-02-16T09:39:59Z

+
+    // "Partition Values"
+    val partitionSchema = partitionTable.partitionSchema()
+    val normalizedSpec = normalizePartitionSpec(


This one is not needed. The job should be done by ResolvePartitionSpec.resolvePartitionSpec, or there is some reason to bypass it?

Yes, the latest version has eliminated the above logic.

MaxGekk · 2023-02-16T09:40:26Z

+    requireExactMatchedPartitionSpec(identifier.toString,
+      normalizedSpec, partitionSchema.fieldNames)
+
+    val partitionNames = normalizedSpec.keySet


Where is it used?

MaxGekk · 2023-02-16T09:43:44Z

+      convertToPartIdent(normalizedSpec, partitionSchema))
+    val partitionIdentifiers = partitionTable.listPartitionIdentifiers(names.toArray, ident)
+    partitionIdentifiers.length match {
+      case 0 =>


The functions extendedPartition() is invoked only for non-empty partition spec as I can see, or not? Is there any test for this case?

MaxGekk · 2023-02-16T09:44:10Z

+      case 0 =>
+        throw QueryExecutionErrors.notExistPartitionError(
+          identifier.toString, ident, partitionSchema)
+      case len if (len > 1) =>


nit:

Suggested change

case len if (len > 1) =>

case len if len > 1 =>

MaxGekk · 2023-02-16T09:49:21Z

+    var i = 0
+    while (i < len) {


This loop:

var i = 0 while (i < len) { i += 1 }

can be simplified by:

for (i <- 0 until len) { }

cloud-fan · 2023-11-02T15:04:01Z

+      catalog: String,
+      namespace: String,
+      table: String): (String, Map[String, String]) = {
+    ("_LEGACY_ERROR_TEMP_1251",


how hard it is to unify this error between v1 and v2 tables?

_LEGACY_ERROR_TEMP_1231(QueryCompilationErrors#invalidPartitionColumnKeyInTableError), there are approximately 13 code point to call it:

spark/common/utils/src/main/resources/error/error-classes.json

Lines 4815 to 4819 in 1359c13

"_LEGACY_ERROR_TEMP_1231" : {

"message" : [

"<key> is not a valid partition column in table <tblName>."

]

},

_LEGACY_ERROR_TEMP_1251(QueryCompilationErrors#actionNotAllowedOnTableSincePartitionMetadataNotStoredError), there are approximately 8 code point to call it:

spark/common/utils/src/main/resources/error/error-classes.json

Lines 4860 to 4864 in 1359c13

"_LEGACY_ERROR_TEMP_1251" : {

"message" : [

"<action> is not allowed on <tableName> since its partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table <tableName>`."

]

},

Because there are many scopes involved, in order to reduce the interference of logic on this PR, I suggest doing the merging or unification in a new separate PR.
Do you think this is appropriate?

make senses, let's do it in followup

cloud-fan

I think it's pretty close now, thanks for your patience!

panbingkun · 2023-11-02T22:08:34Z

I think it's pretty close now, thanks for your patience!

I will update it today.
Thank you very much for your patience and carefulness, which has given me great help! ❤️

Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

cloud-fan · 2023-11-06T06:23:14Z

+    val globalTempViews = if (dbName == globalTempViewManager.database) {
+      globalTempViewManager.listViewNames(pattern).map { viewName =>
+        globalTempViewManager.get(viewName).map(_.tableMeta).getOrElse(
+          throw new NoSuchTableException(globalTempViewManager.database, viewName))


It's weird to throw this error during listing views. Shall we use flatMap and just skip the temp views that were deleted immediately after globalTempViewManager.listViewNames?

cloud-fan · 2023-11-06T06:23:28Z

+
+    val localTempViews = listLocalTempViews(pattern).map { viewIndent =>
+      tempViews.get(viewIndent.table).map(_.tableMeta).getOrElse(
+        throw new NoSuchTableException(viewIndent.database.getOrElse(""), viewIndent.table))


cloud-fan · 2023-11-06T06:24:27Z

      ctx: ShowTableExtendedContext): LogicalPlan = withOrigin(ctx) {
-    val partitionKeys = Option(ctx.partitionSpec).map { specCtx =>
-      UnresolvedPartitionSpec(visitNonOptionalPartitionSpec(specCtx), None)
+    @inline def createUnresolvedTable(


not this kind of inline... we can remove this function and put the code in where we call the function

Okay，I misunderstood the meaning, haha

beliefer · 2023-11-06T12:23:01Z

+    val dbName = format(db)
+    val globalTempViews = if (dbName == globalTempViewManager.database) {


Suggested change

val dbName = format(db)

val globalTempViews = if (dbName == globalTempViewManager.database) {

val globalTempViews = if (format(db) == globalTempViewManager.database) {

beliefer · 2023-11-11T04:11:05Z

+    if (table.supportsPartitions && table.asPartitionable.partitionSchema().nonEmpty) {
+      partitionColumns = table.asPartitionable.partitionSchema()
+      results.put("Partition Provider", "Catalog")
+      results.put("Partition Columns", table.asPartitionable.partitionSchema().map(


Suggested change

results.put("Partition Columns", table.asPartitionable.partitionSchema().map(

results.put("Partition Columns", partitionColumns.map(

cloud-fan · 2023-11-12T09:29:02Z

unfortunately this has conflicts now...

panbingkun · 2023-11-12T13:02:03Z

unfortunately this has conflicts now...

Done, I have resolved these conflicts.

panbingkun · 2023-11-16T01:08:22Z

@cloud-fan If you have time, could you please take a look at this PR? Thank you very much!

cloud-fan · 2023-11-16T05:14:52Z

thanks, merging to master!

panbingkun · 2023-11-16T05:44:16Z

thanks, merging to master!

Thank for all reviewing and great help again @cloud-fan @MaxGekk @beliefer @LuciferYang ❤️❤️❤️

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in DSv2

39f3c42

github-actions Bot added the SQL label Aug 20, 2022

panbingkun changed the title ~~[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2~~ [Don't Review][TEST][SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2 Aug 20, 2022

panbingkun added 2 commits August 20, 2022 21:28

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

cbe7433

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v

023414b

panbingkun changed the title ~~[Don't Review][TEST][SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2~~ [SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2 Aug 20, 2022

MaxGekk requested changes Aug 22, 2022

View reviewed changes

Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala Outdated

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

0697773

panbingkun commented Aug 28, 2022

View reviewed changes

Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala

panbingkun requested a review from MaxGekk August 28, 2022 10:57

panbingkun commented Aug 28, 2022

View reviewed changes

Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala

panbingkun commented Aug 28, 2022

View reviewed changes

Comment thread sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/v2ResolutionPlans.scala Outdated

Fokko mentioned this pull request Nov 28, 2022

show table extended not supported for v2 table. apache/iceberg#5782

Closed

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

676455e

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

9876120

MaxGekk reviewed Feb 16, 2023

View reviewed changes

panbingkun added 3 commits February 20, 2023 17:08

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

7d71eb0

Merge branch 'master' into v2_SHOW_TABLE_EXTENDED

5c40c7c

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

3c3651a

panbingkun requested a review from MaxGekk February 20, 2023 11:45

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

b1cc56c

cloud-fan reviewed Nov 2, 2023

View reviewed changes

Comment thread sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowTablesSuiteBase.scala Outdated

cloud-fan reviewed Nov 2, 2023

View reviewed changes

panbingkun and others added 3 commits November 3, 2023 09:47

Apply suggestions from code review

b47e619

Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

be5b337

Merge branch 'master' into v2_SHOW_TABLE_EXTENDED

73ff15b

panbingkun requested a review from cloud-fan November 3, 2023 13:12

cloud-fan reviewed Nov 6, 2023

View reviewed changes

panbingkun added 2 commits November 6, 2023 19:51

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

e6978e3

Merge branch 'master' into v2_SHOW_TABLE_EXTENDED

1bab140

github-actions Bot removed the BUILD label Nov 6, 2023

panbingkun requested a review from cloud-fan November 6, 2023 12:02

beliefer reviewed Nov 6, 2023

View reviewed changes

fix minor

26ae679

panbingkun mentioned this pull request Nov 10, 2023

[SPARK-45880][SQL] Make the like pattern semantics used in all commands consistent #43751

Closed

beliefer reviewed Nov 11, 2023

View reviewed changes

[SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

ef4213a

Trigger build

d318f6b

cloud-fan approved these changes Nov 16, 2023

View reviewed changes

cloud-fan closed this in c91dbde Nov 16, 2023

cccs-jc mentioned this pull request Mar 17, 2025

[Bug] Improve dbt-spark table listing dbt-labs/dbt-adapters#862

Open

11 tasks

	"_LEGACY_ERROR_TEMP_1231" : {
	"message" : [
	"<key> is not a valid partition column in table <tblName>."
	]
	},

	"_LEGACY_ERROR_TEMP_1251" : {
	"message" : [
	"<action> is not allowed on <tableName> since its partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table <tableName>`."
	]
	},

		val dbName = format(db)
		val globalTempViews = if (dbName == globalTempViewManager.database) {

	val dbName = format(db)
	val globalTempViews = if (dbName == globalTempViewManager.database) {
	val globalTempViews = if (format(db) == globalTempViewManager.database) {

	results.put("Partition Columns", table.asPartitionable.partitionSchema().map(
	results.put("Partition Columns", partitionColumns.map(

Uh oh!

Conversation

panbingkun commented Aug 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

panbingkun commented Aug 20, 2022

Uh oh!

AmplabJenkins commented Aug 21, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fokko commented Nov 28, 2022

Uh oh!

ja-michel commented Nov 30, 2022

Uh oh!

rshanmugam1 commented Dec 2, 2022

Uh oh!

MaxGekk commented Feb 1, 2023

Uh oh!

panbingkun commented Feb 5, 2023

Uh oh!

xkrogen commented Feb 13, 2023

Uh oh!

MaxGekk commented Feb 13, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Aug 20, 2022 •

edited

Loading

panbingkun commented Nov 2, 2023 •

edited

Loading

panbingkun commented Nov 16, 2023 •

edited

Loading

panbingkun commented Nov 16, 2023 •

edited

Loading