retrying problematic files (borg2)

[ThomasWaldmann]

Quite often, especially since borg 1.2 warns about "file changed while it was backed up", there was the wish for a "retry feature" for somehow "failed" files.

Error conditions the might be solvable by retrying:

• file changed while we backed it up (file in use)
• misc. OSError (like I/O errors) (sw or hw issues)
• race conditions with the file type and the file inode number (due to an active fs)

Retrying wasn't easy to implement yet, but I recently made a change to master branch making it simpler:

https://github.com/borgbackup/borg/pull/7349/files - this basically cleans up content chunks incref'd (written) for a file that has been aborted due to an issue (BackupOSError).

In general, we have these scenarios:

• problems before starting to write content chunks to the repo and before writing the file metadata item to the repo: no special cleanup needed, we could just retry.
• problems while reading file contents (IOError == OSError, getting transformed into BackupOSError) and before writing the file metadata item to the repo - cleanup needed.
• problems detected after finishing with backing up the file contents and before writing the file metadata item to the repo (e.g. when borg checks stat of the file again and notices that the file has changed while we read it).
• problems detected after we wrote the file metadata item to the repo: too late, we can not abort that item any more, we could only abort the backup completely.

Note that if we run into troubles with a file (1st try), then clean up the repo from the content chunks, then do the 2nd attempt (1st retry), this might re-transfer content chunks to the repo (which we just had cleaned up). This could be optimized later.

[ThomasWaldmann]

See PR #7351.

[ThomasWaldmann]

This is the log output for a problematic file2, which fails all retries:

A input/file1
A input/file2
read: [Errno 5] simulated I/O error, slept 0.001s, next: retry: 1 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 0.003s, next: retry: 2 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 0.010s, next: retry: 3 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 0.032s, next: retry: 4 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 0.100s, next: retry: 5 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 0.316s, next: retry: 6 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 1.000s, next: retry: 7 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 3.162s, next: retry: 8 of 9...
A input/file2
read: [Errno 5] simulated I/O error, slept 10.000s, next: retry: 9 of 9...
A input/file2
read: [Errno 5] simulated I/O error, giving up (skipping this file).
input/file2: read: [Errno 5] simulated I/O error
E input/file2
A input/file3

The reason for the lines starting with "A" (liked added) is because borg >= 1.2 announces regular files before backing up their content. Only in case of an error, there will be also an "E" line for the same file.

Theoretically the duplicate A input/file2 lines could be suppressed, but some of the error conditions might change that "A" to something else, so guess we better do not do that.

An always-changing file would look like file2, but the final log line would be C input/file2 and borg would back up the file in the last try (it can be more or less corrupted internally).

[ThomasWaldmann]

Please also check the different error handling implemented by the PR depending on the kind of error.

[ThomasWaldmann]

Fixed by #7351.