Skip to content

add any_match higher-order function#21903

Merged
comphead merged 8 commits into
apache:mainfrom
LiaCastaneda:lia/support-for-any-match
Apr 30, 2026
Merged

add any_match higher-order function#21903
comphead merged 8 commits into
apache:mainfrom
LiaCastaneda:lia/support-for-any-match

Conversation

@LiaCastaneda
Copy link
Copy Markdown
Contributor

@LiaCastaneda LiaCastaneda commented Apr 28, 2026 •
edited
Loading

Which issue does this PR close?

  • Closes #.

Rationale for this change

array_any_match is a commonly supported higher-order function in systems like Spark (exists), Trino (any_match) among other engines. It seems like a natural first addition alongside array_transform and worth upstreaming I think.

What changes are included in this PR?

Adds array_any_match(array, predicate) as a new higher-order function (with aliases any_match and list_any_match). It returns:

true if any element satisfies the predicate
false if no element does (including empty arrays)
null if the predicate returns null for some elements and false for all others

Are these changes tested?

Yes I added unit tests and sqllogic tests

Are there any user-facing changes?

Yes -- new SQL functions array_any_match, any_match, and list_any_match are available.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Apr 28, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Apr 28, 2026
@LiaCastaneda LiaCastaneda marked this pull request as ready for review April 28, 2026 21:26
@LiaCastaneda LiaCastaneda mentioned this pull request Apr 28, 2026
Open
27 tasks
comphead
comphead reviewed Apr 28, 2026
View reviewed changes
Comment thread datafusion/sqllogictest/test_files/higher_order.slt Outdated

# predicate returns null for some elements but true for others -> true
query B
SELECT array_any_match([1, NULL, 10], x -> x > 5);
Copy link
Copy Markdown
Contributor

@comphead comphead Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice to add more tests with nulls

select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end)
select exists(array(1, 2, 3), x-> null)

Copy link
Copy Markdown
Contributor Author

@LiaCastaneda LiaCastaneda Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, we cannot yet add a test for
select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end)
because column capture is not yet supported

@LiaCastaneda LiaCastaneda force-pushed the lia/support-for-any-match branch from 7ea1120 to 392b52b Compare April 29, 2026 10:43
@ologlogn
Copy link
Copy Markdown

ologlogn commented Apr 29, 2026 •
edited
Loading

👋 @LiaCastaneda PR #21895 (currently open) adds array_filter and extracts shared HOF utilities into lambda_utils.rs. If that merges first, a few things here could be simplified:

  • coerce_value_types body → crate::lambda_utils::coerce_single_list_arg(self.name(), arg_types)
  • lambda_parameters body → crate::lambda_utils::single_list_lambda_parameters(self.name(), fields)
  • inline take_function_args + destructure in invoke_with_args → crate::lambda_utils::value_lambda_pair(self.name(), &args.args)

@LiaCastaneda
Copy link
Copy Markdown
Contributor Author

LiaCastaneda commented Apr 29, 2026

From @benbellick 's suggestion I verified we are following the substrait semantics/ conventions: https://github.com/substrait-io/substrait/blob/19716584708f98b06eb3f0d73a1f62e15bf6b169/tests/cases/list/any_match.test

comphead
comphead reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THanks @LiaCastaneda the PR looks good to me overall, only nit:
lets rename higher_order.slt into array_any_match.slt

@LiaCastaneda
Copy link
Copy Markdown
Contributor Author

LiaCastaneda commented Apr 30, 2026

I splitted the tests for each function into different sqllogic test files in datafusion/sqllogictest/test_files/array

comphead
comphead approved these changes Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @LiaCastaneda its LGTM

@comphead comphead added this pull request to the merge queue Apr 30, 2026
Merged via the queue into apache:main with commit e18b1cf Apr 30, 2026
36 checks passed
@comphead comphead mentioned this pull request May 5, 2026
Draft
@LiaCastaneda LiaCastaneda mentioned this pull request May 18, 2026
Merged
@LiaCastaneda LiaCastaneda mentioned this pull request May 26, 2026
Merged
LiaCastaneda added a commit to LiaCastaneda/datafusion that referenced this pull request May 26, 2026
@LiaCastaneda @blaginin
add any_match higher-order function (apache#21903)
3afc264 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes #.

## Rationale for this change

`array_any_match` is a commonly supported higher-order function in
systems like Spark (`exists`), Trino (`any_match`) among other engines.
It seems like a natural first addition alongside `array_transform` and
worth upstreaming I think.


## What changes are included in this PR?

Adds `array_any_match(array, predicate)` as a new higher-order function
(with aliases `any_match` and `list_any_match`). It returns:

`true` if any element satisfies the predicate
`false` if no element does (including empty arrays)
`null` if the predicate returns null for some elements and false for all
others

## Are these changes tested?

Yes I added unit tests and sqllogic tests

## Are there any user-facing changes?

Yes -- new SQL functions array_any_match, any_match, and list_any_match
are available.

---------

Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>
LiaCastaneda added a commit to DataDog/datafusion that referenced this pull request May 26, 2026
@LiaCastaneda @blaginin
add any_match higher-order function (apache#21903) (#128)
e0f3307 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #.

## Rationale for this change

`array_any_match` is a commonly supported higher-order function in
systems like Spark (`exists`), Trino (`any_match`) among other engines.
It seems like a natural first addition alongside `array_transform` and
worth upstreaming I think.


## What changes are included in this PR?

Adds `array_any_match(array, predicate)` as a new higher-order function
(with aliases `any_match` and `list_any_match`). It returns:

`true` if any element satisfies the predicate
`false` if no element does (including empty arrays)
`null` if the predicate returns null for some elements and false for all
others

## Are these changes tested?

Yes I added unit tests and sqllogic tests

## Are there any user-facing changes?

Yes -- new SQL functions array_any_match, any_match, and list_any_match
are available.

---------

Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@comphead comphead comphead approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@LiaCastaneda @ologlogn @comphead @blaginin