Skip to content

[DRAFT] enable union backend#439

Merged
Sankara-Jefferson merged 1 commit into
data-preservation-programs:mainfrom
parkan:main
Jul 10, 2024
Merged

[DRAFT] enable union backend#439
Sankara-Jefferson merged 1 commit into
data-preservation-programs:mainfrom
parkan:main

Conversation

@parkan
Copy link
Copy Markdown
Collaborator

@parkan parkan commented Jul 10, 2024

this leverages built-in rclone features to allow backends that have one logical root (e.g. internetarchive) to seamlessly aggregate multiple paths into a single preparation

example usage:

RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''

this creates an implicit, ephemeral remote of the internetarchive type called ia and aggregates paths specified by --upstreams

this leverages built-in rclone features to allow backends that have one logical root (e.g. `internetarchive`) to seamlessly aggregate multiple paths into a single preparation

example usage:
```
RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''
```

this creates an implicit, ephemeral remote of the `internetarchive` type called `ia`  and aggregates paths specified by `--upstreams`
@parkan
Copy link
Copy Markdown
Collaborator Author

parkan commented Jul 10, 2024

why this is a DRAFT: the functionality works as designed but is somewhat specific to the internetarchive remote type as it relies on the implicit creation of a "root" remote (RCLONE_CONFIG_IA_TYPE=internetarchive) on which the upstreams build

for other backend types this becomes potentially non-trivial as the entire remote configuration (auth etc) needs to be specified in env vars (RCLONE_CONFIG_*) and furthermore unions across heterogeneous remote types or other services are untested

all that being said we would really like to see this merged if possible as it is required for SPs to consume preparations created with union sources, to the best of my understanding

the main impediment in my mind is the docstring, which is currently automagically populated from the rclone library; amending it with clarifying comments regarding the above limitations may be sufficient to greenlight this, depending on the maintainer's views, but I'm not entirely sure how to do that

@ianconsolata
Copy link
Copy Markdown
Collaborator

I believe the commits needed to fix the tests can all be pulled from my fork: https://github.com/ianconsolata/singularity/commits/main/. Unfortunately, without maintainer rights I can't update this PR myself.

@Sankara-Jefferson Sankara-Jefferson merged commit a9d3438 into data-preservation-programs:main Jul 10, 2024
Sankara-Jefferson pushed a commit that referenced this pull request Jul 10, 2024
These are some additional changes I made on a separate fork that cover
the errors introduced in #439, which should make the main build green
again.
Sankara-Jefferson pushed a commit that referenced this pull request Jun 14, 2025
this leverages built-in rclone features to allow backends that have one
logical root (e.g. `internetarchive`) to seamlessly aggregate multiple
paths into a single preparation

example usage:
```
RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''
```

this creates an implicit, ephemeral remote of the `internetarchive` type
called `ia` and aggregates paths specified by `--upstreams`
Sankara-Jefferson pushed a commit that referenced this pull request Jun 14, 2025
These are some additional changes I made on a separate fork that cover
the errors introduced in #439, which should make the main build green
again.
anjor pushed a commit to anjor/singularity that referenced this pull request Jun 14, 2025
this leverages built-in rclone features to allow backends that have one
logical root (e.g. `internetarchive`) to seamlessly aggregate multiple
paths into a single preparation

example usage:
```
RCLONE_CONFIG_IA_TYPE=internetarchive  singularity storage create union --upstreams 'ia:DuckandC1951 ia:0872_Angels_Flight_05_32_34_00' --path ''
```

this creates an implicit, ephemeral remote of the `internetarchive` type
called `ia` and aggregates paths specified by `--upstreams`
anjor pushed a commit to anjor/singularity that referenced this pull request Jun 14, 2025
…ation-programs#439 (data-preservation-programs#441)

These are some additional changes I made on a separate fork that cover
the errors introduced in data-preservation-programs#439, which should make the main build green
again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants