Spark: Add separate data and metadata file lists to RewriteTablePath#15382
Spark: Add separate data and metadata file lists to RewriteTablePath#15382mxm wants to merge 5 commits into
Conversation
| String fileListLocation = saveFileList(copyPlan, RESULT_LOCATION); | ||
| String dataFileListLocation = saveFileList(dataCopyPlan, DATA_FILE_LIST_DIR); |
There was a problem hiding this comment.
Don't we afraid of the cost of duplicated file list write?
There was a problem hiding this comment.
We potentially write a lot of files (metadata, manifest lists, manifests), so writing two instead of one txt file looks ok to me.
| @Override | ||
| public RewriteContentFileResult append(RewriteResult<ContentFile<?>> r1) { | ||
| this.copyPlan().addAll(r1.copyPlan()); | ||
| this.toRewrite().addAll(r1.toRewrite()); | ||
|
|
||
| public RewriteContentFileResult append(RewriteContentFileResult other) { | ||
| this.copyPlan().addAll(other.copyPlan()); | ||
| this.toRewrite().addAll(other.toRewrite()); |
There was a problem hiding this comment.
The actual return type from where this is being called it RewriteContentFileResult, so I decided to narrow the type.
RewriteResult relies on generics which are subject to type erasure, see
It could be worthwhile to get rid of the types, since the different types aren't treated any different.
mxm
force-pushed
the
rewrite-table-path-file-lists
branch
from
6f88d67 to
9577ecd
Compare
|
Rebased to resolve conflicts with #15381. |
mxm
force-pushed
the
rewrite-table-path-file-lists
branch
from
9577ecd to
356bebf
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
I'm keeping this open for now. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
2 participants
Downstream tools often need different copy strategies for data vs metadata files (e.g. different location, additional checks on data vs metadata files). This splits the file list into separate data and metadata lists while keeping the combined list for backward compatibility.