Skip to content

[SPARK-48307][SQL][FOLLOWUP] not-inlined CTE references sibling should not fail#47141

Closed
cloud-fan wants to merge 2 commits into
apache:masterfrom
cloud-fan:fix
Closed

[SPARK-48307][SQL][FOLLOWUP] not-inlined CTE references sibling should not fail#47141
cloud-fan wants to merge 2 commits into
apache:masterfrom
cloud-fan:fix

Conversation

@cloud-fan

@cloud-fan cloud-fan commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This is a follow-up of #46617 to fix a bug. When we re-construct the WithCTE node, we should use the new CTE definitions that have been applied inlineCTE.

Why are the changes needed?

bug fix, otherwise we may hit errors such as

java.util.NoSuchElementException: key not found: 0
	at scala.collection.MapOps.default(Map.scala:289)
	at scala.collection.MapOps.default$(Map.scala:288)
	at scala.collection.AbstractMap.default(Map.scala:420)
	at scala.collection.mutable.HashMap.apply(HashMap.scala:440)
	at org.apache.spark.sql.catalyst.optimizer.PushdownPredicatesAndPruneColumnsForCTEDef$.gatherPredicatesAndAttributes(PushdownPredicatesAndPruneColumnsForCTEDef.scala:74)
	at org.apache.spark.sql.catalyst.optimizer.PushdownPredicatesAndPruneColumnsForCTEDef$.$anonfun$gatherPredicatesAndAttributes$1(PushdownPredicatesAndPruneColumnsForCTEDef.scala:68)
	at scala.collection.immutable.Vector.foreach(Vector.scala:2124)
	at org.apache.spark.sql.catalyst.optimizer.PushdownPredicatesAndPruneColumnsForCTEDef$.gatherPredicatesAndAttributes(PushdownPredicatesAndPruneColumnsForCTEDef.scala:67)

Does this PR introduce any user-facing change?

no, the bug is not released yet

How was this patch tested?

new test

Was this patch authored or co-authored using generative AI tooling?

no

@github-actions github-actions Bot added the SQL label Jun 28, 2024
plan match {
case WithCTE(child, cteDefs) =>
val remainingDefs = cteDefs.filter { cteDef =>
val notInlined = mutable.ArrayBuffer.empty[CTERelationDef]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also the behavior of the previous code.

Comment thread sql/core/src/test/scala/org/apache/spark/sql/CTEInlineSuite.scala Outdated
@cloud-fan

Copy link
Copy Markdown
Contributor Author

}

test("SPARK-48307: not-inlined CTE references sibling") {
withTempView("t") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find where you create or use this view

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I copied it from other test but didn't end up using it. Let me remove

@cloud-fan

Copy link
Copy Markdown
Contributor Author

thanks for the review, merging to master!

@cloud-fan cloud-fan closed this in 4e57f06 Jun 28, 2024

@amaliujia amaliujia left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late LGTM thanks!

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants