Skip to content

[SPARK-51695][SQL][DOCS][FOLLOW-UP] Clarify NULL handling semantics in UNIQUE constraint Javadoc #54357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yyanyy wants to merge 1 commit into from
Closed

[SPARK-51695][SQL][DOCS][FOLLOW-UP] Clarify NULL handling semantics in UNIQUE constraint Javadoc #54357

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/Unique.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@
* <p>
* A UNIQUE constraint specifies one or more columns as unique columns. Such constraint is satisfied
* if and only if no two rows in a table have the same non-null values in the unique columns.
* Unlike PRIMARY KEY, UNIQUE allows nullable columns.
* <p>
* NULL values are treated as distinct from each other (NULLS DISTINCT semantics). Two rows

@gengliangwang gengliangwang Feb 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don’t need to define the suggested behavior in Spark itself. The behavior can be left to the connector implementations.

@holdenk holdenk Feb 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the flipside if the connectors are different then querying the same data from two different backends could result in different results which might be confusing. Although I doubt the backends are going to care about our constraint spec here (although the connectors, hopefully, if they can't follow it would drop it).

@gengliangwang gengliangwang Feb 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, it makes sense to clarify the expected behavior for NULL semantics on the Spark side. This will help users and developers understand whether a constraint can be declared as RELY.

@yyanyy yyanyy Feb 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you both for the review! I think in future for extensibility, we could define an API to have this default overridden per connector, so that a future optimizer rule may look this up and take advantage of it on a per connector/table basis if needed, though it's indeed will result in a quite complicated logic.

* are considered duplicates only when every column in the unique key has a non-null value and
* every value matches. If any column in the unique key is NULL, the row is always considered
* unique regardless of other values. In other words, multiple rows with NULL in one or more
* unique columns are allowed and do not violate the constraint definition.
* <p>
* The {@code NULLS NOT DISTINCT} modifier is not currently supported.
* <p>
* Spark doesn't enforce UNIQUE constraints but leverages them for query optimization. Each
* constraint is either valid (the existing data is guaranteed to satisfy the constraint), invalid
Expand Down