[SPARK-58221][SQL] Add try_variant_set expression - #57373
Closed
bojana-db wants to merge 4 commits into
Closed
Conversation
bojana-db
force-pushed
the
try-variant-set
branch
from
July 20, 2026 10:43
28474b7 to
1125feb
Compare
bojana-db
changed the title
try_variant_set expressiontry_variant_set expression
harshmotw-db
approved these changes
Jul 21, 2026
harshmotw-db
left a comment
harshmotw-db
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks for the feature!
srielau
approved these changes
Jul 21, 2026
bojana-db
force-pushed
the
try-variant-set
branch
from
July 23, 2026 12:48
7a13a70 to
3c2da74
Compare
Member
|
Thank you @bojana-db and @harshmotw-db @srielau! Proceeding to merge. |
uros-b
approved these changes
Jul 24, 2026
uros-b
changed the title
try_variant_set expressiontry_variant_set expression
uros-b
pushed a commit
that referenced
this pull request
Jul 24, 2026
### What changes were proposed in this pull request? Adds the SQL function `try_variant_set(v, path, val[, create_if_missing])`, the error-tolerant counterpart of `variant_set` (SPARK-57804). It sets (inserts or replaces) a value in a Variant value at a single JSONPath location, returning NULL instead of failing the query when the operation hits a recoverable error. Details: - On a valid set, behaves exactly like `variant_set`: an object path (e.g. `$.a`) replaces the field if it exists and creates it when `create_if_missing` is true (the default); an array path (e.g. `$[N]`) replaces the element at index `N`, and when `N` is at or past the end and `create_if_missing` is true, pads the array with variant nulls up to `N`; missing intermediate keys/indices along the path are created when `create_if_missing` is true; - When `create_if_missing` is false, a missing leaf, a missing intermediate, or an out-of-range array index leaves `v` unchanged; - Recoverable errors are caught and return NULL instead of throwing: any path type mismatch (`VARIANT_PATH_TYPE_MISMATCH`, e.g. an object key applied to an array); - Non-recoverable errors still throw, matching `variant_set`: a malformed path, including the root path `$`, is rejected with `INVALID_VARIANT_PATH` (the path is parsed before the set), and a result exceeding the size limit raises `VARIANT_SIZE_LIMIT`; - Any NULL argument returns NULL; - The value may be any expression castable to variant. ### Why are the changes needed? Error-tolerant counterpart of `variant_set`, matching `try_variant_get`, `try_variant_insert`, and other `try_*` functions. ### Does this PR introduce _any_ user-facing change? Yes, a new SQL function `try_variant_set` (and the corresponding Scala/Python `functions` API). ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor with Claude Opus 4.8 Closes #57373 from bojana-db/try-variant-set. Authored-by: bojana-db <bojana.zecevic@databricks.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit 759fbd9) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds the SQL function
try_variant_set(v, path, val[, create_if_missing]), the error-tolerant counterpart ofvariant_set(SPARK-57804). It sets (inserts or replaces) a value in a Variant value at a single JSONPath location, returning NULL instead of failing the query when the operation hits a recoverable error.Details:
variant_set: an object path (e.g.$.a) replaces the field if it exists and creates it whencreate_if_missingis true (the default); an array path (e.g.$[N]) replaces the element at indexN, and whenNis at or past the end andcreate_if_missingis true, pads the array with variant nulls up toN; missing intermediate keys/indices along the path are created whencreate_if_missingis true;create_if_missingis false, a missing leaf, a missing intermediate, or an out-of-range array index leavesvunchanged;VARIANT_PATH_TYPE_MISMATCH, e.g. an object key applied to an array);variant_set: a malformed path, including the root path$, is rejected withINVALID_VARIANT_PATH(the path is parsed before the set), and a result exceeding the size limit raisesVARIANT_SIZE_LIMIT;Why are the changes needed?
Error-tolerant counterpart of
variant_set, matchingtry_variant_get,try_variant_insert, and othertry_*functions.Does this PR introduce any user-facing change?
Yes, a new SQL function
try_variant_set(and the corresponding Scala/PythonfunctionsAPI).How was this patch tested?
Unit tests.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor with Claude Opus 4.8