Seanaye/persistence trait

[seanaye]

Description

This MR introduces a Persistence trait which abstracts away the persistence layer for Completed bao files. This means that you can BYO persistence impl to the current fs store (e.g. S3).

Breaking Changes

The external facing breaking changes are mostly around the "asyncification" of certain trait methods. I will update this section in more detail later.

Notes & open questions

This is still a draft MR I am just looking for feedback on the general approach before I clean things up a bit more. e.g. There are some places that can panic right now

Change checklist

• Self-review.
• Documentation updates following the style guide, if relevant.
• Tests if relevant.
• All breaking changes documented.

[rklaehn]

Thanks for the PR. I will have to check if the storage abstraction adds performance costs for the very important local/sync case.

It might be that the best way to deal with this use case is to have a separate store implementation that allows plugging in an async storage backend, since the local store is very performance critical.

The way I usually test this is by using sendme and then syncing a directory structure that contains a lot of tiny files, like the linux kernel.

Warning: I got some important deadlines the next weeks and won't be able to look into this until after may 18.

[seanaye]

Thanks for taking a look, I will test with sendme. Is there a specific command you use to benchmark the impl? I still have a few commits to push

[matthiasbeyer]

I believe this cannot be trivially rebased, right?

But I think this is crucial and should not be neglected by the iroh authors! Implementing the storage backend yourself is what downstream projects want to do most likely!

[okdistribute]

Hi @seanaye, are you still interested in using iroh-blobs for your work? This PR cannot be trivially rebased but we are interested in picking it back up again, maybe a quick discussion on what you want to accomplish and we can see how we can get it done

[seanaye]

I could look into picking this back up again. It would be nice to support async storage for wasm contexts. I'm not sure what the timeline would look like on my end though

[matthiasbeyer]

async storage would also be nice for non-wasm contexts, because persistence is IO and doing that async'ly is (IMHO) the way to go. Especially because streaming is then only another step (streaming objects from network to storage, which would have the potential of greatly reducing memory load).