create: implement retries for individual fs files - #7351
Merged
ThomasWaldmann merged 1 commit intoFeb 23, 2023
Conversation
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## master #7351 +/- ##
==========================================
- Coverage 83.98% 83.93% -0.05%
==========================================
Files 67 67
Lines 11692 11707 +15
Branches 2126 2131 +5
==========================================
+ Hits 9819 9826 +7
- Misses 1312 1317 +5
- Partials 561 564 +3
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
ThomasWaldmann
force-pushed
the
retry-failed-files
branch
from
February 14, 2023 10:46
d5549ec to
1aa7d76
Compare
ThomasWaldmann
changed the title
sophie-h
reviewed
Feb 14, 2023
ThomasWaldmann
force-pushed
the
retry-failed-files
branch
from
February 17, 2023 19:21
4a04660 to
65771ae
Compare
Errors handled for backup src files: - BackupOSError (converted from OSError), e.g. I/O Error - BackupError (stats race, file changed while we backed it up) Error Handling: - retry the same file after some sleep time - sleep time starts from 1ms, increases exponentially up to 10s - 10 tries If retrying does not help: - BackupOSError: skip the file, log it with "E" status - BackupError: last try will back it up, log it with "C" status Works for: - borg create's normal (builtin) fs recursion - borg create --paths-from-command - borg create --paths-from-stdin Notes: - update stats.files_stats late (so we don't get wrong stats in case of e.g. IOErrors while reading the file). - _process_any: no changes to the big block, just indented for adding the retry loop and the try/except. - test_create_erroneous_file succeeds because we retry the file.
ThomasWaldmann
force-pushed
the
retry-failed-files
branch
from
February 23, 2023 00:19
dc491db to
7e6afc9
Compare
|
any chance this one could be backported to 1.4.x ? |
Member
Author
|
@drzraf Originally, I did not intend to backport this. Can you tell me your use case, so I maybe can re-evaluate? |
Contributor
|
My observation is that file changes happen very frequently on desktop backups. SQLite dbs for messengers, browser databases for bookmarks etc |
Exactly this. (I've been a very satisfied user of Borg for years, but this particular output/behavioral tweak is what I've awaiting the most :)) |
Member
Author
ThomasWaldmann
added a commit
to ThomasWaldmann/borg
that referenced
this pull request
Jul 5, 2026
…ckup#7351) Backport of borgbackup#7351 to 1.4-maint. Retry reading a backup source file when it raises a BackupError / BackupOSError (e.g. transient I/O error, or the file changed while we read it): - retry the same file after some sleep time (1ms, growing exponentially up to 10s, 10 tries) - BackupOSError: if retries do not help, skip the file, log it with "E" status - BackupError (changed while reading): last try backs it up, logged with "C" status Works for borg create's builtin fs recursion, --paths-from-command and --paths-from-stdin. 1.4-specific: do NOT retry a file once checkpoint part files (.borg_part_N) have been written for it. Re-reading it from the start would create duplicate / inconsistent part files (concatenating all part files would then no longer yield the complete file). ChunksProcessor.last_part_number tracks this; such a file is kept as-is ("C") or skipped ("E") instead of being retried. Permission errors (EPERM/EACCES) are not retried (they won't get better), so we don't waste ~15s of backoff on every unreadable file. Adds a ChunkerFailing test chunker (chunker-params=fail,block_size,map) to drive the error handling, and tests: - test_create_erroneous_file: file is read on a later retry - test_create_no_permission_file: permission error is not retried, file skipped - test_create_erroneous_file_with_part_files: file with part files is not retried Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ThomasWaldmann
added a commit
to ThomasWaldmann/borg
that referenced
this pull request
Jul 7, 2026
…ckup#7351) Backport of borgbackup#7351 to 1.4-maint. Retry reading a backup source file when it raises a BackupError / BackupOSError (e.g. transient I/O error, or the file changed while we read it): - retry the same file after some sleep time (1ms, growing exponentially up to 10s, 10 tries) - BackupOSError: if retries do not help, skip the file, log it with "E" status - BackupError (changed while reading): last try backs it up, logged with "C" status Works for borg create's builtin fs recursion, --paths-from-command and --paths-from-stdin. 1.4-specific: do NOT retry a file once checkpoint part files (.borg_part_N) have been written for it. Re-reading it from the start would create duplicate / inconsistent part files (concatenating all part files would then no longer yield the complete file). ChunksProcessor.last_part_number tracks this; such a file is kept as-is ("C") or skipped ("E") instead of being retried. Permission errors (EPERM/EACCES) are not retried (they won't get better), so we don't waste ~15s of backoff on every unreadable file. Adds a ChunkerFailing test chunker (chunker-params=fail,block_size,map) to drive the error handling, and tests: - test_create_erroneous_file: file is read on a later retry - test_create_no_permission_file: permission error is not retried, file skipped - test_create_erroneous_file_with_part_files: file with part files is not retried Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Errors handled for backup src files:
Error Handling:
If retrying does not help:
Works for:
Notes:
stats in case of e.g. IOErrors while reading the file).
for adding the retry loop and the try/except.