In ARROW-14908 we solved the case of joining a dataset to an in memory table, but did not solve joining two datasets.
The previous solution was to add +1 to the thread count, because the hash join logic might be called by the scanner's IO thread. For joining more than 1 dataset, we might have more than 1 IO thread, so we either need to add a larger arbitrary number or find a way to make the state logic more resilient to unexpected threads.
Reporter: Will Jones / @wjones127
Assignee: Will Jones / @wjones127
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-15718. Please see the migration documentation for further details.
In ARROW-14908 we solved the case of joining a dataset to an in memory table, but did not solve joining two datasets.
The previous solution was to add +1 to the thread count, because the hash join logic might be called by the scanner's IO thread. For joining more than 1 dataset, we might have more than 1 IO thread, so we either need to add a larger arbitrary number or find a way to make the state logic more resilient to unexpected threads.
Reporter: Will Jones / @wjones127
Assignee: Will Jones / @wjones127
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-15718. Please see the migration documentation for further details.