[C++] arrow-compute-hash-join-node-test failed

[asfimport]

arrow-compute-hash-join-node-test failed in my environment (windows, msvc 19.12.25835)

because in HashJoinBasicImpl::ProbeBatch_Lookup hash_table_ is empty, but hash_table_empty_ is false. 

I workaround this issue using the attached patch, but fix likely should be different.

Reporter: Renat Valiullin / @rip-nsk
Assignee: Michal Nowakiewicz / @michalursa

Related issues:

• [C++] Occasional failure arrow-compute-hash-join-node-test (is related to)

Original Issue Attachments:

• workaround.patch

PRs and other links:

• GitHub Pull Request #12894

_{Note: This issue was originally created as ARROW-14911. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
CC @michalursa @save-buffer

[asfimport]

Sasha Krassovsky / @save-buffer:
Hi [~rip.nsk@gmail.com]

Could you please share the parameters you gave to the join and what data you used?

Thanks!

[asfimport]

Renat Valiullin / @rip-nsk:
@save-buffer,

I don't know how to get this data, is there way to save shuffled_input_arrays into file?

TEST(HashJoin, Random) prints

RIGHT_SEMI IS_IS int[0]int[0]
payload_l: str[7..36]int[1]str[13..38]
payload_r: int[1]str[12..15]
num_rows_l = 17242 num_rows_r = 16832 batch size = 75 parallel = true
before HashJoinWithExecPlan fails.

BTW, workaround patch does not help.

[asfimport]

Antoine Pitrou / @pitrou:
@save-buffer @michalursa Can one of you perhaps create a draft branch or PR with some additional logging to help diagnosing what happens on [~rip.nsk@gmail.com]'s setup?

Also, it seems the code is almost uncommented and undocumented in hash_join.(h|cc), it would be nice if you could create a PR to improve readability and maintainability there.

[asfimport]

Sasha Krassovsky / @save-buffer:
One thing is that this code will be gone in a few weeks (Michal is rewriting hash join completely). For now I'll try to reproduce it.

[asfimport]

Weston Pace / @westonpace:
I'm not sure the input data will be as important. If I am reading the test correctly it initializes a random generator with a fixed seed. So even though the test is testing random data it should be deterministic between runs. I have verified that we are running this test on our CI servers (which includes VS2017 although at a higher version than 19.12).

The test does do some parallel execution so there is a possibility it is a race condition causing the flakiness.

I also built locally with VS2019 in both debug mode and release and tried running this test both with and without CPU stress and have not been able to reproduce it. I was doing this against the latest master and not the 6.0.1 tag so that might be a factor. I will try and reproduce this some more tomorrow with VS2017.

[~rip.nsk@gmail.com] How reliable is this failure? Does it fail every time or intermittently? When it fails does it always display the exact same output? (e.g. RIGHT SEMI IS_IS...)

At the moment I'm not sure this qualifies as a blocker.

[asfimport]

Weston Pace / @westonpace:
I ran tests against the 6.0.1 tag without any failures. I then downgraded to VS 2017 but I was not able to get the exact same version. Microsoft only makes versions 19.10 and 19.15 available for download. 19.10 fails to compile and I am unable to reproduce the failure with 19.15.

If the failure is reliable it's possible that there is a hardware mismatch. The join node makes heavy use of intrinsics and although they are guarded by ARROW_HAVE_AVX2 there might be some odd corner case.

[~rip.nsk@gmail.com] Do you know what kind of CPU you are running on?

I'm going to move this from blocker to major.

[asfimport]

Krisztian Szucs / @kszucs:
Postponing it to 8.0 then.

[asfimport]

Weston Pace / @westonpace:
Issue resolved by pull request 12894
#12894