Skip to content

[SPARK-54417][SQL] Fix error message for scalar subquery in IDENTIFIER clause #53133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mihailotim-db wants to merge 1 commit into from
Closed

[SPARK-54417][SQL] Fix error message for scalar subquery in IDENTIFIER clause #53133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,12 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString
errorClass = "UNSUPPORTED_FEATURE.OVERWRITE_BY_SUBQUERY",
messageParameters = Map.empty)

case p: PlanWithUnresolvedIdentifier if !p.identifierExpr.resolved =>
p.identifierExpr.failAnalysis(
errorClass = "NOT_A_CONSTANT_STRING.NOT_CONSTANT",

@amaliujia amaliujia Nov 19, 2025 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we can have better error message. Per my understanding the IDENTIFIER can take:

  • relation (table or view) name
  • function name
  • column name
  • field name
  • schema name
  • catalog name

The error NOT_A_CONSTANT_STRING.NOT_CONSTANT is too general as which is not a constant expression whose equivalent value is known at query planning time.

Because this is a very specific scenario, I am thinking if we can create a new error message to say IDENTIFIER xxx is not table, view, function, column, field, schema or catalog name

@mihailotim-db mihailotim-db Nov 20, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with adding a new error message in the format you propose is that we create a dependency for available features. So if we add to this list, we would also have to update the error message.

That being said, I was thinking of creating a separate exception for this case but they all ended up being something like Couldn't resolve identifier, identifier needs to be a constant string. So I decided to leave this exception as the same one is thrown from IdentifierResolution.evalIdentifierExpr

messageParameters = Map("name" -> "IDENTIFIER", "expr" -> p.identifierExpr.sql)
)

case operator: LogicalPlan =>
operator transformExpressionsDown {
case hof: HigherOrderFunction if hof.arguments.exists {
Expand Down
118 changes: 118 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/identifier-clause-legacy.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,124 @@ org.apache.spark.sql.AnalysisException
}


-- !query
CREATE TABLE t(col1 INT)
-- !query analysis
CreateDataSourceTableCommand `spark_catalog`.`default`.`t`, false


-- !query
SELECT * FROM IDENTIFIER((SELECT 't'))
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 26,
"stopIndex" : 37,
"fragment" : "(SELECT 't')"
} ]
}


-- !query
SELECT * FROM (SELECT IDENTIFIER((SELECT 'col1')) FROM IDENTIFIER((SELECT 't')))
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 67,
"stopIndex" : 78,
"fragment" : "(SELECT 't')"
} ]
}


-- !query
SELECT IDENTIFIER((SELECT 'col1')) FROM VALUES(1)
-- !query analysis
org.apache.spark.sql.AnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 19,
"stopIndex" : 33,
"fragment" : "(SELECT 'col1')"
} ]
}


-- !query
SELECT col1, IDENTIFIER((SELECT col1)) FROM VALUES(1)
-- !query analysis
org.apache.spark.sql.AnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery(col1)",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 25,
"stopIndex" : 37,
"fragment" : "(SELECT col1)"
} ]
}


-- !query
SELECT IDENTIFIER((SELECT 'col1', 'col2')) FROM VALUES(1,2)
-- !query analysis
org.apache.spark.sql.catalyst.parser.ParseException
{
"errorClass" : "UNSUPPORTED_TYPED_LITERAL",
"sqlState" : "0A000",
"messageParameters" : {
"supportedTypes" : "\"DATE\", \"TIMESTAMP_NTZ\", \"TIMESTAMP_LTZ\", \"TIMESTAMP\", \"INTERVAL\", \"X\", \"TIME\"",
"unsupportedType" : "\"SELECT\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 20,
"stopIndex" : 32,
"fragment" : "SELECT 'col1'"
} ]
}


-- !query
DROP TABLE t
-- !query analysis
DropTable false, false
+- ResolvedIdentifier V2SessionCatalog(spark_catalog), default.t


-- !query
CREATE TABLE IDENTIFIER(1)(c1 INT) USING csv
-- !query analysis
Expand Down
118 changes: 118 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/identifier-clause.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,124 @@ org.apache.spark.sql.AnalysisException
}


-- !query
CREATE TABLE t(col1 INT)
-- !query analysis
CreateDataSourceTableCommand `spark_catalog`.`default`.`t`, false


-- !query
SELECT * FROM IDENTIFIER((SELECT 't'))
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 26,
"stopIndex" : 37,
"fragment" : "(SELECT 't')"
} ]
}


-- !query
SELECT * FROM (SELECT IDENTIFIER((SELECT 'col1')) FROM IDENTIFIER((SELECT 't')))
-- !query analysis
org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 67,
"stopIndex" : 78,
"fragment" : "(SELECT 't')"
} ]
}


-- !query
SELECT IDENTIFIER((SELECT 'col1')) FROM VALUES(1)
-- !query analysis
org.apache.spark.sql.AnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery()",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 19,
"stopIndex" : 33,
"fragment" : "(SELECT 'col1')"
} ]
}


-- !query
SELECT col1, IDENTIFIER((SELECT col1)) FROM VALUES(1)
-- !query analysis
org.apache.spark.sql.AnalysisException
{
"errorClass" : "NOT_A_CONSTANT_STRING.NOT_CONSTANT",
"sqlState" : "42601",
"messageParameters" : {
"expr" : "scalarsubquery(col1)",
"name" : "IDENTIFIER"
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 25,
"stopIndex" : 37,
"fragment" : "(SELECT col1)"
} ]
}


-- !query
SELECT IDENTIFIER((SELECT 'col1', 'col2')) FROM VALUES(1,2)
-- !query analysis
org.apache.spark.sql.catalyst.parser.ParseException
{
"errorClass" : "UNSUPPORTED_TYPED_LITERAL",
"sqlState" : "0A000",
"messageParameters" : {
"supportedTypes" : "\"DATE\", \"TIMESTAMP_NTZ\", \"TIMESTAMP_LTZ\", \"TIMESTAMP\", \"INTERVAL\", \"X\", \"TIME\"",
"unsupportedType" : "\"SELECT\""
},
"queryContext" : [ {
"objectType" : "",
"objectName" : "",
"startIndex" : 20,
"stopIndex" : 32,
"fragment" : "SELECT 'col1'"
} ]
}


-- !query
DROP TABLE t
-- !query analysis
DropTable false, false
+- ResolvedIdentifier V2SessionCatalog(spark_catalog), default.t


-- !query
CREATE TABLE IDENTIFIER(1)(c1 INT) USING csv
-- !query analysis
Expand Down
8 changes: 8 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/identifier-clause.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,14 @@ VALUES(IDENTIFIER(1));
VALUES(IDENTIFIER(SUBSTR('HELLO', 1, RAND() + 1)));
SELECT `IDENTIFIER`('abs')(c1) FROM VALUES(-1) AS T(c1);

CREATE TABLE t(col1 INT);
SELECT * FROM IDENTIFIER((SELECT 't'));
SELECT * FROM (SELECT IDENTIFIER((SELECT 'col1')) FROM IDENTIFIER((SELECT 't')));
SELECT IDENTIFIER((SELECT 'col1')) FROM VALUES(1);
SELECT col1, IDENTIFIER((SELECT col1)) FROM VALUES(1);
SELECT IDENTIFIER((SELECT 'col1', 'col2')) FROM VALUES(1,2);
DROP TABLE t;

CREATE TABLE IDENTIFIER(1)(c1 INT) USING csv;
CREATE TABLE IDENTIFIER('a.b.c')(c1 INT) USING csv;
CREATE VIEW IDENTIFIER('a.b.c')(c1) AS VALUES(1);
Expand Down
Loading