Optimize the concat_ws function#3869
Merged
andygrove merged 2 commits intoOct 18, 2022
Merged
Conversation
Signed-off-by: remzi <13716567376yh@gmail.com>
github-actions
Bot
added
logical-expr
Signed-off-by: remzi <13716567376yh@gmail.com>
alamb
approved these changes
Oct 18, 2022
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
This is beautiful @HaoYang670
| } | ||
| } | ||
|
|
||
| impl Literal for &String { |
Comment on lines
+325
to
+354
| for arg in args { | ||
| match arg { | ||
| // filter out null args | ||
| Expr::Literal(ScalarValue::Utf8(None) | ScalarValue::LargeUtf8(None)) => {} | ||
| Expr::Literal(ScalarValue::Utf8(Some(v)) | ScalarValue::LargeUtf8(Some(v))) => { | ||
| match contiguous_scalar { | ||
| None => contiguous_scalar = Some(v.to_string()), | ||
| Some(mut pre) => { | ||
| pre += delimiter; | ||
| pre += v; | ||
| contiguous_scalar = Some(pre) | ||
| } | ||
| } | ||
| } | ||
| Expr::Literal(s) => return Err(DataFusionError::Internal(format!("The scalar {} should be casted to string type during the type coercion.", s))), | ||
| // If the arg is not a literal, we should first push the current `contiguous_scalar` | ||
| // to the `new_args` and reset it to None. | ||
| // Then pushing this arg to the `new_args`. | ||
| arg => { | ||
| if let Some(val) = contiguous_scalar { | ||
| new_args.push(lit(val)); | ||
| } | ||
| new_args.push(arg.clone()); | ||
| contiguous_scalar = None; | ||
| } | ||
| } | ||
| } | ||
| if let Some(val) = contiguous_scalar { | ||
| new_args.push(lit(val)); | ||
| } |
Contributor
There was a problem hiding this comment.
This pattern of creating the contiguous scalar is so similar -- I wonder if it could be extracted out into a function -- perhaps as a follow on PR
Contributor
Author
There was a problem hiding this comment.
Thank you for reviewing @alamb
The logic for concat and concat_ws is a little different, because in concat_ws we must consider the delimiter and we can't ignore the empty string literals. I will try to find a way to refactor them.
| args: new_args, | ||
| } | ||
| } | ||
| } => simpl_concat(args)?, |
| // the delimiter is not a literal | ||
| { | ||
| let expr = concat_ws(col("c"), vec![lit("a"), null.clone(), lit("b")]); | ||
| let expected = concat_ws(col("c"), vec![lit("a"), lit("b")]); |
Member
|
Thanks @HaoYang670 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: remzi 13716567376yh@gmail.com
Which issue does this PR close?
Closes #3856.
Closes #3857.
Rationale for this change
Simplify the
concat_wsexpression:nullif the delimiter is nullnullargumentsconcatto replaceconcat_wsif the delimiter is an empty stringWhat changes are included in this PR?
Are there any user-facing changes?