feat: new backup transfer method by dignifiedquire · Pull Request #3489 · chatmail/core

dignifiedquire · 2022-07-07T21:56:42Z

⚠️ ~~There be dragons~~ 🐉
There be exciting things

Idea: use p2p based transfer methods to move backups from one device to the other.

Implementation: Use iroh a new implementation of IPFS to build iroh-share, a simple protocol that moves data from device a to device b.

Local Testing

Using the new commands send-backup and receive-backup <Ticket> you can test this on a local machine or between two machines running the repl example.

Current Restrictions

Both devices must be in the same network to discover each other

TODOs

Before Merging

Future

improve iroh-shares hole punching so it does not need to be on the same network anymore
add python bindings
add node bindings

dignifiedquire · 2022-07-07T22:07:17Z

    pub fn suffix(&self) -> Option<&str> {
        let ext = self.name.rsplit('.').next();
-        if ext == Some(&self.name) {
+        if ext == Some(self.name.as_str()) {


I had to make these changes because things would suddenly not compile anymore otherwise, not sure what happened there, seems like a compiler issue 🤷

Simon-Laux · 2022-07-10T14:04:10Z

In which direction does this go? only "scan to import" for now?
Though I guess it's probably easy to switch directions once the connection works at all (to also allow scan code from computer (when computer has no camera) to export from phone to that computer)

link2xt · 2022-09-10T17:46:54Z

librocksdb takes forever to build, why is it needed as a dependency? Is it possible to disable it as a feature somehow?
I see iroh supports different stores: https://github.com/n0-computer/iroh/tree/main/stores

flub · 2022-09-11T10:10:29Z

+ *     The path must exist and being accessible at least until the other device has received all data.
+ *     The folder is not cleaned up by the backup send;
+ *     a temporary directory seems to be good place therefore.


I'm not a UI developer, but this seems a bit awkward and something that the core could all handle internally?

I don't have strong opinions in either direction @r10s what do you think?

in general, it is nicer if the UI would not need to take care about such details.

but not sure how easy it really is for the core to really fulfill the requirements. so, as a pragmatic approach, i am also fine with leaving things up to the UIs for now, where we can more safely pass a directly that is really handled by the OS.

flub · 2022-09-11T10:11:48Z

+ *     The path must exist and being accessible at least until the other device has received all data.
+ *     The folder is not cleaned up by the backup send;
+ *     a temporary directory seems to be good place therefore.
+ * @param passphrase Used to at-rest-encrypt the backuped database in `folder` before sending,


Suggested change

* @param passphrase Used to at-rest-encrypt the backuped database in `folder` before sending,

* @param passphrase Used to at-rest-encrypt the backup in `folder` before sending,

no this is correct, only the database is encrypted

flub · 2022-09-11T10:13:31Z

+ *     similar to dc_imex().
+ *     The same passphrase must be given to dc_receive_backup() on the other device.
+ *     If NULL or empty string is given, the sent backup is not encrypted at rest.
+ *     The transport of the backup is always encrypted additionally.


If the transport is encrypted why is this passphrase still needed? Would be nice if this described what it is protecting against as well.

This passphrase is there to encrypt the sql database. It is the same as the existing other backup methods. I personally would love to drop it, but didn't want to open that can of worms.

yeah, i'd also better leave that can of worms closed, at least for now :)

flub · 2022-09-11T10:24:59Z

+
+
+/**
+ * Waits for the sending to finish and frees the backup sender object.


Is thee a way to abort the sending?

Or resume any time later (incl. never)?

flub · 2022-09-11T10:28:19Z

+ * but on a device that has scanned the QR code generated by dc_backup_sender_qr().
+ *
+ * While dc_receive_backup() returns immediately, the started job may take a while;
+ * you can stop it using dc_stop_ongoing_process().


An ongoing process is IIRC a global modal that can not be dismissed. I guess this is normally done from the "create account" screen or similar so that is what is desired?

dignifiedquire · 2022-09-13T18:10:02Z

librocksdb takes forever to build, why is it needed as a dependency? Is it possible to disable it as a feature somehow?
I see iroh supports different stores: https://github.com/n0-computer/iroh/tree/main/stores

@link2xt no it is not optional, those stores are just for experimentation. RocksDB is the only supported DB in iroh atm

flub

foo-shm and foo-wal are checked in accidentally i think.

CI isn't happy yet, but i think this is probably mostly fine

dignifiedquire · 2022-09-19T19:47:17Z

foo-shm and foo-wal are checked in accidentally i think.

ups, fixed

…ng`"

* add dc_receive_backup ffi * fix compile issues * make clippy happy Co-authored-by: dignifiedquire <me@dignifiedquire.com>

…al parameters

dignifiedquire · 2022-11-23T15:54:31Z

iroh requires rust1.63, so this bumps the msrv for dc

r10s · 2023-03-06T18:17:41Z

closing, this is superseded by #4007

dignifiedquire mentioned this pull request Jul 7, 2022

[WIP] feat: add new backup transfer mechanism deltachat/deltachat-ios#1643

Closed

dignifiedquire commented Jul 7, 2022

View reviewed changes

r10s mentioned this pull request Jul 9, 2022

add DCLOGIN uri scheme deltachat/interface#48

Merged

dignifiedquire force-pushed the iroh-share branch from bd5156e to 73d10d4 Compare July 11, 2022 14:42

This was referenced Jul 11, 2022

Iroh share docs #3500

Merged

[WIP] Iroh share UI deltachat/deltachat-ios#1649

Closed

dignifiedquire force-pushed the iroh-share branch from cc4b9bd to 33e9aeb Compare July 28, 2022 14:58

dignifiedquire force-pushed the iroh-share branch from 33e9aeb to 634f4c2 Compare August 29, 2022 12:39

dignifiedquire changed the title ~~[WIP] feat: new backup transfer method~~ feat: new backup transfer method Aug 29, 2022

dignifiedquire requested review from link2xt and r10s August 29, 2022 12:46

dignifiedquire marked this pull request as ready for review August 29, 2022 13:03

dignifiedquire force-pushed the iroh-share branch from 8a66efb to a8c2a31 Compare August 31, 2022 08:50

r10s mentioned this pull request Sep 1, 2022

iroh resp. rocksdb does not compile on android #3570

Closed

flub reviewed Sep 11, 2022

View reviewed changes

Comment thread src/imex.rs

Comment thread src/imex.rs

Comment thread src/imex.rs Outdated

Comment thread src/imex.rs Outdated

Comment thread src/imex.rs

Comment thread src/imex.rs

Comment thread src/imex.rs

Comment thread src/imex.rs Outdated

Comment thread src/imex.rs

dignifiedquire force-pushed the iroh-share branch from a8c2a31 to edeb36d Compare September 19, 2022 09:18

flub approved these changes Sep 19, 2022

View reviewed changes

dignifiedquire force-pushed the iroh-share branch from 5da980e to 9701078 Compare October 5, 2022 14:46

r10s mentioned this pull request Oct 8, 2022

Multi-Device Setup for easier onboarding / QR + noise #1731

Closed

dignifiedquire added 6 commits November 23, 2022 16:25

pull in iroh-share

0e06bcb

integrate backup sending via iroh into the repl

9f7e962

implement receive-backup

14fc7b7

update deps

364e06d

remove unused event

679242d

generate qr codes

63872fc

dignifiedquire and others added 18 commits November 23, 2022 16:27

proper shutdown and progress reports

0aa4e21

fix "the trait Add<&&str> is not implemented for `std::string::Stri…

4ff746e

…ng`"

repl: documentation and tab-completion for the new send-backup commands

94c16ef

fix: make passphrase work for repl's send-backup

17f01e5

document ffi wrt sending and receiving backups

e61ee76

add dc_receive_backup ffi (#3504)

0ed3348

* add dc_receive_backup ffi * fix compile issues * make clippy happy Co-authored-by: dignifiedquire <me@dignifiedquire.com>

dc_reveive_backup() ffi just takes the qr-code again instead of sever…

411d0a7

…al parameters

clear config cache before receiving a backup (with new config values)

1633b2f

update iroh-share & cleanup api

f696f3e

apply updates

6c6499f

ci: protoc fix (#3566)

ba869a2

update deps

09d5cbe

apply CR

26719ee

fixup

1f4384b

clippy fixes

c9cd4c9

fixup rebase

b2c68c7

clippy

b6e8c9f

fixup

1fd4e8e

dignifiedquire force-pushed the iroh-share branch from 83b2453 to 1fd4e8e Compare November 23, 2022 15:36

dignifiedquire added 2 commits November 23, 2022 16:50

update dependencies, including iroh to 0.1.1

415279c

update github workflows

2056452

dignifiedquire added 2 commits December 2, 2022 16:51

Merge remote-tracking branch 'origin/master' into iroh-share

0bcba98

update to released iroh version

94b50f6

hpk42 mentioned this pull request Dec 4, 2022

don't lowercase blob extensions #3227

Closed

ramfox mentioned this pull request Feb 17, 2023

iroh-share is merged into DeltaChat n0-computer/beetle#17

Open

r10s mentioned this pull request Mar 1, 2023

Add DcJsonrpcInstance class deltachat/deltachat-android#2479

Merged

2 tasks

r10s closed this Mar 6, 2023

r10s mentioned this pull request Mar 6, 2023

Send and receive backup over network using QR code #4007

Merged

8 tasks

	* @param passphrase Used to at-rest-encrypt the backuped database in `folder` before sending,
	* @param passphrase Used to at-rest-encrypt the backup in `folder` before sending,



		/**
		* Waits for the sending to finish and frees the backup sender object.

Uh oh!

Conversation

dignifiedquire commented Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Local Testing

Current Restrictions

TODOs

Before Merging

Future

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Simon-Laux commented Jul 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

link2xt commented Sep 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r10s Sep 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dignifiedquire commented Sep 13, 2022

Uh oh!

flub left a comment

Choose a reason for hiding this comment

Uh oh!

dignifiedquire commented Sep 19, 2022

Uh oh!

dignifiedquire commented Nov 23, 2022

Uh oh!

r10s commented Mar 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dignifiedquire commented Jul 7, 2022 •

edited

Loading

Simon-Laux commented Jul 10, 2022 •

edited

Loading

link2xt commented Sep 10, 2022 •

edited

Loading

r10s Sep 23, 2022 •

edited

Loading