IN LIST: optimize Utf8View and BinaryView filters - #24088
Draft
geoffreyclaude wants to merge 8 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24088 +/- ##
==========================================
+ Coverage 80.91% 81.15% +0.23%
==========================================
Files 1102 1114 +12
Lines 377102 387354 +10252
Branches 377102 387354 +10252
==========================================
+ Hits 305143 314345 +9202
- Misses 53769 54461 +692
- Partials 18190 18548 +358 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 5, 2026 07:02
ec9cb2e to
65b67ba
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listCPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 6, 2026 16:01
65b67ba to
a44ee0d
Compare
geoffreyclaude
changed the title
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 9, 2026 06:21
a44ee0d to
b0443b6
Compare
10 tasks
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
10 times, most recently
from
August 11, 2026 16:29
61cc7a6 to
156d76b
Compare
github-actions
Bot
added
auto detected api change
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
2 times, most recently
from
August 12, 2026 10:18
6a5284e to
f8ae3ad
Compare
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (f8ae3ad) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Contributor
|
I noticed this is marked as draft -- please ping me when ready to review |
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
7 times, most recently
from
August 13, 2026 10:14
854fa0b to
dab222a
Compare
Use PrimitiveHashSetFilter<T, K> for integer and floating-point arrays. Integer filters keep their native key type by default; Float32 and Float64 use their existing bitwise wrapper keys, preserving signed-zero and NaN-payload behavior. The key type and conversion are selected statically, with no function pointer or dynamic dispatch in the lookup loop. Filter selection, dictionary handling, null handling, IN, and NOT IN behavior remain unchanged. Decimal128 routing remains in the following commit.
Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 13, 2026 10:46
dab222a to
cc8aacf
Compare
Treat inline Utf8View and BinaryView values as 128-bit primitive keys and pass them to the shared primitive filter selector. This reuses direct comparisons for short lists and the primitive hash-set path for larger lists without reading backing buffers. Lists containing a non-inline value continue to use the general filter. Exact view types, dictionaries, slices, nulls, IN, and NOT IN keep their existing behavior.
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 13, 2026 11:42
cc8aacf to
00dd083
Compare
Contributor
Author
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 00dd083 (00dd083) to 7ff49f9 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"
changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 00dd083 (00dd083) to 7ff49f9 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "7ff49f94d14f73151a55f59314247c4c223073f2"
changed:
ref: "00dd083e02da959679b1b5996f9b64cafe2cff75"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Note
Until #24102 merges, GitHub's Files changed tab also shows the earlier PRs in the stack. The direct diff from #24102 to this PR shows only this PR's byte-view changes.
Rationale for this change
Arrow represents every
Utf8ViewandBinaryViewvalue with a 16-byte view. When a value is at most 12 bytes long, the complete value is stored directly in that view:Equal short values therefore have the same 128-bit view. DataFusion can look up that view directly, without reading another buffer or using Arrow's general value comparison.
This PR uses that faster path when every non-null value in the
INlist is at most 12 bytes long. The byte-view adapter treats each inline view as aDecimal128key and passes it to the shared primitive selector from #24283, the same selector used by #24102:Decimal128is only a 16-byte container here; no decimal operations are performed.A longer input value cannot match a short list value because its encoded length is different, so it can be rejected without reading its backing bytes. If the list itself contains a value longer than 12 bytes, DataFusion keeps using the general filter.
Null list entries do not affect this choice, but are still recorded for SQL null behavior. After unwrapping any dictionary, the input and list must use the same view type:
Utf8Viewis not mixed withBinaryView, and regularUtf8andBinarykeep their existing paths.What changes are included in this PR?
Utf8ViewandBinaryView.IN, andNOT IN.Are these changes tested?
Tests cover both view types and both faster paths; the 4/5-value and 12/13-byte boundaries; long inputs and lists that require the general filter; exact-type routing; slices and dictionaries; input and list nulls; and
INandNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Automated benchmark snapshot
Automated run on an aarch64 Neoverse V2 runner, comparing #24102 with this PR at exact commit
00dd083e. The benchmark evaluates 8,192-row batches after filter construction. Lower is better.All 15 reported rows improved, with a 35.9% geometric-mean time reduction.
utf8view/short_8b/list=4/match=0%utf8view/short_8b/list=4/match=50%utf8view/short_8b/list=16/match=0%utf8view/short_8b/list=16/match=50%utf8view/short_8b/list=64/match=0%utf8view/short_8b/list=64/match=50%utf8view/short_8b/list=256/match=0%utf8view/short_8b/list=256/match=50%utf8view/len_12b/list=16/match=0%utf8view/len_12b/list=16/match=50%utf8view/len_12b/list=64/match=0%utf8view/len_12b/list=64/match=50%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_INnulls/utf8view/short_8b/list=16/match=50%/nulls=50%