fix: split "union" and "interleave"

[crepererum]

Which issue does this PR close?

Fixes #5970.

This is an alternative to #6009. Closes #6009

Rationale for this change

UnionExec shall be a simple node and not perform some optimizations on its own. If repartition / partition concatenation is desired, then there should be an optimizer pass for that. Being a simple node avoids confusion and optimizer bugs like #5970.

What changes are included in this PR?

Remove the entire "partition aware" logic from UnionExec

Are these changes tested?

1. Existing tests pass
2. Test for UNION ALL with ORDER BY results are inconsistent #5970 passes.
3. Query mentioned in his comment is correctly optimized (i.e. does not require top-level repartition)

Are there any user-facing changes?

Less bugs.

[mingmwang]

Could you please add a UT for the rule itself ?
For example for the case there are multiple partition-aware UnionExecs, finally they are all replaced.

[mingmwang]

I'm afraid the current approach will not work in below case, additional Hash Repartition will still be added:

select count(*) from {
select distinct name from table t1
union all
select distinct name from table t2
) group by name

[crepererum]

will add a unit test for the optimizer. will also try to add a test for the query listed above. I think this can be fixed by pushing the Interleave optimizer rule before Repartition.

[crepererum]

I have this working locally by changing the optimizer. Instead of having an extra optimizer, this has to be integrated into the "dist enforcement" transform-up system. I'll need to add more tests though and clean up the code, but I think the result is satisfying.

[mingmwang]

I have this working locally by changing the optimizer. Instead of having an extra optimizer, this has to be integrated into the "dist enforcement" transform-up system. I'll need to add more tests though and clean up the code, but I think the result is satisfying.

Yes, this should works, thanks!

[mingmwang]

LGTM, except for some wrong comments.

[ozankabak]

Does the new InterleaveExec behave similar to UnionExec in terms of sort enforcement? There are a couple places in sort_enforcement.rs where it makes a call to is_union to optimize/enforce sorts efficiently. Do we need to make any changes there?

@mustafasrepo, PTAL

[mingmwang]

InterleaveExec

InterleaveExec can not keep the ordering(maintains_input_order is false), I think the logic in the rules do not need to update.

[mingmwang]

I think this PR can be merged to main.

[alamb]

Thanks @mingmwang