Skip to content

ARROW-15498: [C++][Compute] Implement Bloom filter pushdown between hash joins #12289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
save-buffer wants to merge 42 commits into from
Closed

ARROW-15498: [C++][Compute] Implement Bloom filter pushdown between hash joins #12289

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
7cbafc8
Adding Bloom filter implementation
michalursa Jan 3, 2022
012175a
Add Bloom Filter Pushdown
save-buffer Jan 25, 2022
e529280
Fix after rebase
save-buffer Apr 12, 2022
3b75380
clang-format
save-buffer Apr 12, 2022
0e8a770
Standardize on int64_t, make rngs thread-local
save-buffer Apr 13, 2022
244aaf1
Delete some random unnecessary code, fix rand_int
save-buffer Apr 14, 2022
a3dfc3b
Make thread sanitizer happy
save-buffer Apr 14, 2022
df5f8cd
Make Bloom filter test multithreaded. This upsets tsan.
save-buffer Apr 14, 2022
eb8117e
Dumb bug
save-buffer Apr 20, 2022
48204fd
Remove my thread fences
save-buffer Apr 20, 2022
6839829
Rebase
save-buffer Apr 21, 2022
05b935c
Switch to int64_t again
save-buffer Apr 21, 2022
7e594b2
Switch to unique_ptr
save-buffer Apr 21, 2022
5ec4e0c
Make bloom filter blocks atomics
save-buffer Apr 21, 2022
4f5dbcd
Switch back to non-atomics, but silence tsan for these functions
save-buffer Apr 22, 2022
c7e250b
clang-format
save-buffer Apr 22, 2022
3af8ce4
clang-format
save-buffer Apr 22, 2022
21eb53a
static_cast
save-buffer Apr 23, 2022
7c58dd1
Use condition variable in test
save-buffer Apr 23, 2022
dbb773f
Make it green
save-buffer Apr 25, 2022
80301a4
Respond to Weston comments
save-buffer Apr 26, 2022
9f67171
ARROW_EXPORT
save-buffer Apr 26, 2022
7aa7458
More ARROW_EXPORT
save-buffer Apr 26, 2022
a10a6a8
Remove line
save-buffer Apr 27, 2022
9f23c88
Pray to the mighty barney that his condition variable may work
save-buffer Apr 29, 2022
54d5022
Add an underscore
save-buffer Apr 29, 2022
f1335e1
Document parameter, explicitly initialize variable in constructor
save-buffer Apr 29, 2022
9929b84
Document another thing
save-buffer Apr 29, 2022
f68ea78
Fix dumb bug
save-buffer Apr 30, 2022
75bf765
Run fewer tests
save-buffer Apr 30, 2022
1aaeb6f
Respond to michal comments
save-buffer May 5, 2022
643eeb5
Fix CV and fix TSAN BloomFilterBuilder (thanks Michal and Weston)
save-buffer May 9, 2022
b28f4d5
Fix on big endian
save-buffer May 9, 2022
b18a7d3
Run fewer tests with ASAN
save-buffer May 9, 2022
caa3193
Fix weston comments
save-buffer May 9, 2022
16e7d76
clang-format
save-buffer May 9, 2022
68c3ba6
Make windows happy again
save-buffer May 9, 2022
bdbcb01
clang-format
save-buffer May 9, 2022
36a229b
Try fixing on big endian again
save-buffer May 16, 2022
6e5bf4c
clang-format
save-buffer May 16, 2022
a3b2f93
Disable bloom filter on big endian
save-buffer May 17, 2022
1a4ae69
Hopefully fix timeout
save-buffer May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Run fewer tests with ASAN
  • Loading branch information
@save-buffer
save-buffer committed May 17, 2022
commit b18a7d3542abee72b7837f5de0aa1b9c0cc89868
2 changes: 1 addition & 1 deletion cpp/src/arrow/compute/exec/bloom_filter.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,7 @@ void BloomFilterBuilder_Parallel::PushNextBatchImp(size_t thread_id, int64_t num

PartitionSort::Eval(
num_rows, num_prtns, partition_ranges,
[hashes, num_prtns](int64_t row_id) {
[hashes, num_prtns, kLogBlocksKeptTogether](int64_t row_id) {
constexpr int kPrtnIdBitOffset =
BloomFilterMasks::kLogNumMasks + 6 + kLogBlocksKeptTogether;
return (hashes[row_id] >> (kPrtnIdBitOffset)) & (num_prtns - 1);
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/compute/exec/hash_join_node.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,7 +608,7 @@ class HashJoinNode : public ExecNode {
}

bool all_comparisons_is = true;
for (JoinKeyCmp cmp : key_cmp_) all_comparisons_is &= cmp == JoinKeyCmp::IS;
for (JoinKeyCmp cmp : key_cmp_) all_comparisons_is &= (cmp == JoinKeyCmp::IS);

if ((join_type_ == JoinType::RIGHT_OUTER || join_type_ == JoinType::FULL_OUTER) &&
all_comparisons_is)
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/exec/hash_join_node_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1000,6 +1000,8 @@ TEST(HashJoin, Random) {
Random64Bit rng(42);
#if defined(THREAD_SANITIZER) || defined(ARROW_VALGRIND)
const int num_tests = 15;
#elsif defined(ADDRESS_SANITIZER)
const int num_tests = 50;
#else
const int num_tests = 100;
#endif
Expand Down