[DRAFT] enable union backend#439
Conversation
this leverages built-in rclone features to allow backends that have one logical root (e.g. `internetarchive`) to seamlessly aggregate multiple paths into a single preparation example usage: ``` RCLONE_CONFIG_IA_TYPE=internetarchive singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path '' ``` this creates an implicit, ephemeral remote of the `internetarchive` type called `ia` and aggregates paths specified by `--upstreams`
|
why this is a DRAFT: the functionality works as designed but is somewhat specific to the for other backend types this becomes potentially non-trivial as the entire remote configuration (auth etc) needs to be specified in env vars ( all that being said we would really like to see this merged if possible as it is required for SPs to consume preparations created with union sources, to the best of my understanding the main impediment in my mind is the docstring, which is currently automagically populated from the rclone library; amending it with clarifying comments regarding the above limitations may be sufficient to greenlight this, depending on the maintainer's views, but I'm not entirely sure how to do that |
|
I believe the commits needed to fix the tests can all be pulled from my fork: https://github.com/ianconsolata/singularity/commits/main/. Unfortunately, without maintainer rights I can't update this PR myself. |
this leverages built-in rclone features to allow backends that have one logical root (e.g. `internetarchive`) to seamlessly aggregate multiple paths into a single preparation example usage: ``` RCLONE_CONFIG_IA_TYPE=internetarchive singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path '' ``` this creates an implicit, ephemeral remote of the `internetarchive` type called `ia` and aggregates paths specified by `--upstreams`
this leverages built-in rclone features to allow backends that have one logical root (e.g. `internetarchive`) to seamlessly aggregate multiple paths into a single preparation example usage: ``` RCLONE_CONFIG_IA_TYPE=internetarchive singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path '' ``` this creates an implicit, ephemeral remote of the `internetarchive` type called `ia` and aggregates paths specified by `--upstreams`
…ation-programs#439 (data-preservation-programs#441) These are some additional changes I made on a separate fork that cover the errors introduced in data-preservation-programs#439, which should make the main build green again.
this leverages built-in rclone features to allow backends that have one logical root (e.g.
internetarchive) to seamlessly aggregate multiple paths into a single preparationexample usage:
this creates an implicit, ephemeral remote of the
internetarchivetype callediaand aggregates paths specified by--upstreams