Core: Improve HadoopFileIO performance when working with cloud storage#15586
Core: Improve HadoopFileIO performance when working with cloud storage#15586steveloughran wants to merge 5 commits into
Conversation
steveloughran
changed the title
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
still live |
steveloughran
force-pushed
the
pr/15353-cloud-io
branch
from
64eadc2 to
fa92607
Compare
* set read policy on filetype. * if status is set, pass in * if length is known, pass in * handle the future completion by extracting the cause
- read policy set in open file TODO - validate policy choice of puffin files
+ roll back making listing closeable(); too much a change for too little value.
steveloughran
force-pushed
the
pr/15353-cloud-io
branch
from
fa92607 to
7d0dd4e
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
This was marked stale and closed while I was on a european length holiday...clearly it is designed for a US developer world where a 2 week break is unusual |
1 participant
Fixes #15353
Improve file opening and read times by
Util.determineReadPolicy()).ParquetIO already hands down file opening to parquet, which does the right thing.l
What matters for it is retaining any FileStatus already obtained, which is what the changes in
TableMigrationUtildo.It's a shame that parquet (currently) lacks a way to skip that stat() call which is does to get file length, as this adds a HEAD request to all openings of a parquet file where the length is known from a manifest. That is fixable and would save 100+mS per file opening, as well as the associated IO capacity.
However, until #12554 is fixed, that manifest file length can't be trusted, so the stat() matters. It's possibly why this issue hasn't been noticed on iceberg-java code