Support Rewrite Datafiles into a custom Partition Spec

[RussellSpitzer]

Feature Request / Improvement

Currently rewrite data-files always uses the current table partition spec when rewriting but it is useful to be able to use any partition spec in the table. In this way, a user could potentially write with one spec like hours, but then rewrite into days.

This would require expanding the API and then changing the implementation a bit. We should probably do this in two PR's, one adding the new API. Then additional follow up PR's actually adding implementations for the various existing rewrites like Spark's Rewrite DataFiles.

Query engine

Spark

[edgarRd]

I added this for the same use case when we had the previous rewrite data files action still in core in PR #2293 but apparently that didn't make it to the new RewriteDataFiles. It's still used in Flink tho.

I agree, It'd be great to consolidate to have only one data files rewrite API.

[matternc]

Do you see this as less about Partition evolution and more about being able to rewrite data into a more optimal partition spec. Like in your example maybe your default partition is using days, but on one specific day your data was exceptionally chatty resulting in suboptimal query performance. So being able to rewrite these chatty days into hours could result in a more optimal partition setup.

If so, would it make sense for users to be able to add a partition spec to a table without altering/evolving the existing default. Maybe this already exists?

[RussellSpitzer]

Note that @dpaani Has a PR for this.
#7585

[himadripal]

here is a revised PR #9803 for this issue for spark 3.5

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'