analyze: deduplicated size of a set of archives, #5741

[ThomasWaldmann]

Implements the request in #5741: report the deduplicated size of a set of archives (e.g. all backups matching -a 'sh:userA-*'), following ThomasWaldmann's proposal there.

Every figure is reported as source (uncompressed source data) and stored (compressed, as stored in the repository) size, plus the compression factor relating the two (stored / source).

Deduplicated size of a set of archives

For the considered (matching) set of archives:

• Deduplicated size of set — summed size of the union of chunks the set references (chunks shared within the set counted once).
• Exclusive size of set — summed size of chunks referenced only by the set and by no other archive, i.e. the space that deleting the whole set would free.
• Unreferenced chunks — chunks that no non-deleted archive references, i.e. what borg compact could free in the current state of the repository. Only their stored size is known: a chunk's source size is only recorded in the archives referencing it, and these have none.

Following the issue comment, chunks are flagged as referenced by the considered set (C) and/or by the rest of the archives (R); the exclusive size is sum(size(x) for x in C - R).

$ borg analyze -a 'sh:web'

Deduplicated size of the 2 considered archive(s)
===================================================================
Considered archives: 2 (of 4 in the repository)

                                  source        stored  compression
Deduplicated size of set:        8.00 MB       3.26 MB         0.41
Exclusive size of set:           2.00 MB     820.62 kB         0.41
Unreferenced chunks:                 n/a           0 B          n/a

Without an archive filter, the considered set is all archives - every referenced chunk is then trivially exclusive to it, so that line is suppressed and the report is labeled as the whole repository.

Decomposition by archive series (--by-series)

Every chunk is counted in exactly one row, so the rows add up to the repository's deduplicated size. This answers "which series costs how much, and what would I get back by dropping it" for all series at once:

$ borg analyze --by-series

Repository decomposition by archive series
=================================================================================
4 archive(s) in 3 series

name                            archives        source        stored  compression
db                                     1       4.00 MB       1.64 MB         0.41
logs                                   1       3.00 MB       1.22 MB         0.41
web                                    2       2.00 MB     820.62 kB         0.41
(shared by 2+ series)                          6.00 MB       2.44 MB         0.41
(unreferenced)                                     n/a           0 B          n/a
---------------------------------------------------------------------------------
total (deduplicated)                   4      15.01 MB       6.11 MB         0.41

This needs only a single pass over the archives: bits 5..22 of the ephemeral user flags hold the series that first referenced a chunk (as index + 1, so 0 means unreferenced), bit 23 marks chunks referenced by more than one series. As the shared and unreferenced rows can only be determined by looking at every archive, --by-series always covers the whole repository and rejects archive filters.

Implementation

• Reuses the repository's own chunk index (repository.chunks) instead of building a second index — it already holds each chunk's stored size (obj_size). The membership flag bits are OR-ed in and the source size (0 in the repo index) is filled in from the per-archive references cache, preserving pack_id/obj_offset/obj_size via _replace. These in-memory mutations never persist: write_chunkindex_to_repo zeroes flags and size, and close() only serializes F_NEW entries (none here).
• Chunk membership and source sizes come from the per-archive references cache maintained by borg compact, so unchanged archives usually do not need to be opened/decrypted.
• The references-cache helpers (get_archive_references & friends, ArchiveReferences) are moved from archiver/compact_cmd.py into cache.py as free functions so analyze and compact share one implementation; compact keeps its previous behavior.

Notes

• The dedup-size report works for one or more matching archives; the existing directory hot-spot report still requires at least two (skipped with an info log otherwise).
• -a needs the sh: prefix for globs (borg2's default match is exact).
• Chunks of soft-deleted archives count as unreferenced (soft-delete keeps the archive's chunks and its reference cache; borg compact only keeps them when the repository is damaged, so borg undelete stays possible).
• Also drops the now-stale deduplicated-size note from the borg info epilog (borg info no longer computes per-archive deduplicated sizes) and points to borg analyze / borg compact --stats instead.

Tests

analyze_cmd_test.py covers: set dedup vs exclusive size with shared and unique files, the single-archive set, the whole-repository (no filter) report, the --by-series decomposition (asserting the rows add up to the total), the rejection of filters with --by-series, and the unreferenced-chunks report - including that borg compact then frees exactly the reported number of objects. Existing analyze/compact/cache/info suites pass.

[codecov]

Codecov Report

❌ Patch coverage is 88.84462% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.79%. Comparing base (a50eb6f) to head (7161a0b).
⚠️ Report is 14 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/archiver/analyze_cmd.py	90.28%	10 Missing and 7 partials ⚠️
src/borg/cache.py	84.50%	9 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9971      +/-   ##
==========================================
- Coverage   85.70%   82.79%   -2.92%     
==========================================
  Files          95       95              
  Lines       16815    17021     +206     
  Branches     2577     2607      +30     
==========================================
- Hits        14411    14092     -319     
- Misses       1668     2191     +523     
- Partials      736      738       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.