Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf#7790
Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf#7790wypoon wants to merge 2 commits into
Conversation
|
I opened another PR also aims to expose some configuration to SQL conf. Seems that Iceberg only allows a small set of configurations to be overwritten by SQL session configuration. I'm wondering if we can expose all configurations to SQL conf. |
|
@pan3793 I think it would be on a case-by-case basis depending on the use case for each configuration. |
wypoon
changed the title
|
I think read configs like |
@manuzhang you can add your voice to the thread I already started on the dev list. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
3 participants
Currently, the write delete/update/merge mode for a table (copy-on-write or merge-on-read) is specified as a table property in the table. For some use cases, it makes sense to want to use copy-on-write for some Spark jobs writing to a table and merge-on-read for other jobs writing to the same table. Constantly toggling the table property is not a usable workaround.
In this change, we allow the mode to be specified in Spark's SQLConf. If it is specified in the SQLConf, that setting overrides what is set in the corresponding table property.