check: keep the records of corrupt packs across check cycles

[mr-raj12]

Refs #9696.

cache/checked-packs was holding two things with different lifetimes in one file: the cycle progress ("already hashed this pass", throwaway) and the set of packs that verified corrupt.

Sharing one lifetime meant the corrupt set never outlived the check that produced it. A completed cycle called tracker.clear(), which empties the table and deletes the store object. The next full check cleared it again before starting. So a cheap cron check had no way to hand its findings to a later repair, and the corrupt pack ids only ever showed up as log lines in whichever run happened to find them.

The fix keeps one file and one format and changes the retention rule instead. Intact records (result=1) stay cycle-scoped and are dropped when a cycle completes. Corrupt records (result=0) are kept until the pack verifies intact again or is no longer listed in packs/.

The cycle-end tracker.clear() becomes tracker.finish_cycle(pack_ids): drop the intact records, prune records of packs that are gone, save what is left or delete the object if nothing is. Pruning is only correct at that point, since a completed cycle is the only moment the full packs/ listing has been seen. A partial check stops early and would prune packs it had not reached yet.

The summary now names the corrupt packs, guarded on index_errors == 0 so the ids only appear when the pack check actually ran.

A separate cache/corrupt-packs object was the other option. I went with the single table because result already encodes the distinction, 9543c14 already treats result=0 specially by re-verifying those packs, and two files can drift out of sync where one cannot.

The existing skip guard re-verifies every result=0 entry, so a full check still hashes 100% of packs and nothing about the checking behaviour changes. A healthy repo ends up byte-identical to before: the table empties, the object is deleted, no extra IO. There is no format change and no version bump, old blobs load unchanged, and an older borg reading a newer blob just sees result=0 entries it re-verifies anyway.

Five tests in repository_test.py:

• corrupt records survive a completed cycle while intact ones are dropped
• the summary names the corrupt ids
• a carried-over corrupt record gets re-verified and dropped once the pack is intact
• a corrupt record for a pack no longer listed is pruned at cycle end
• corrupt records survive across consecutive partial checks

repository_test.py, check_cmd_test.py, cache_test.py and compact_cmd_test.py pass. black and ruff are clean.

Not in here: machine-readable output, --repair consuming the list, clearing records after a successful repair. Those depend on how borg check --repair and a separate repair command end up being split, which is still open.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.52%. Comparing base (5878040) to head (7e5325a).
⚠️ Report is 39 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9925      +/-   ##
==========================================
- Coverage   85.49%   82.52%   -2.97%     
==========================================
  Files          93       93              
  Lines       16122    16147      +25     
  Branches     2466     2476      +10     
==========================================
- Hits        13783    13325     -458     
- Misses       1632     2121     +489     
+ Partials      707      701       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

I think we could keep all entries except the ones for packs that do not exist any more.

See also #9898.

The condition for skipping a pack would then change from "it is in checked-packs AND it was OK" to "it is in checked-packs AND it was OK AND the result is recent enough".

[mr-raj12]

I think we could keep all entries except the ones for packs that do not exist any more.

See also #9898.

The condition for skipping a pack would then change from "it is in checked-packs AND it was OK" to "it is in checked-packs AND it was OK AND the result is recent enough".

Implemented as a new --max-age INTERVAL option (e.g. 4w). Skip condition is now: in checked-packs AND it was OK AND the record is younger than max_age. Intact records are kept across cycles, only records of packs no longer listed in packs/ are pruned at cycle end. Corrupt records are always re-verified.
Default is 0 (current behaviour, full check verifies every pack) - can change it to a non-zero default if you prefer.

[ThomasWaldmann]

CI: /home/runner/work/borg/borg/docs/misc/asciinema/README.rst: WARNING: document isn't included in any toctree [toc.not_included]

I'll fix that, was a collateral damage of my asciinema updates.

Update: #9938 merged, please rebase on current master.