check: track a set of checked packs for partial checks#9909
Open
mr-raj12 wants to merge 3 commits into
Open
Conversation
Replace the single last-pack-checked marker with a persisted set of checked pack ids, so a pack added between partial runs is verified regardless of how its content-sha256 name sorts.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9909 +/- ##
==========================================
+ Coverage 85.13% 85.32% +0.19%
==========================================
Files 93 93
Lines 15899 15926 +27
Branches 2428 2429 +1
==========================================
+ Hits 13535 13589 +54
+ Misses 1654 1633 -21
+ Partials 710 704 -6 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
requested changes
Jul 14, 2026
ThomasWaldmann
left a comment
Member
There was a problem hiding this comment.
Also: does it handle Ctrl-C correctly?
Split the vertical import and trim an irrelevant line from the checked-packs comment.
b9214b6 to
0e68c7e
Compare
Contributor
Author
Yes, store.store() writes atomically and appends a sha256 that's verified on load, so an interrupt can't corrupt the set or mark a pack checked when it wasn't. The worst case you redo a last one minute of work, capped at 1 min now that checkpoints run every 60s. |
Also shrink the partial-check checkpoint window from 5 minutes to 1 minute so a Ctrl-C between checkpoints redoes at most a minute of verification work.
0e68c7e to
996331b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The repository check writes a marker holding the last pack id it verified, so a partial check (
--max-duration) resumes by skipping every pack up to that marker. This assumes the pack listing is sorted.Pack files are named by the sha256 of their content, so a pack added between two partial runs can sort anywhere. If its id sorts before the marker, the resumed check skips it and never verifies it.
This replaces the marker with a persisted set of checked pack ids, a
HashTableNTmapping pack id to(timestamp, result). A pack is skipped only when it appears in the set recorded as intact; a pack recorded as corrupt is re-verified rather than skipped. So a newly added pack gets checked whatever its name sorts to, and a pack that was corrupt in an earlier run is looked at again. The set is dropped once a cycle has checked every pack.The set is managed by a
PackTrackerclass. It persists viastore.store()(atomic write) with a sha256 appended over the serialized table; on load a blob whose sha256 does not match is discarded, so a rotted set re-checks everything instead of skipping an unverified pack. Ctrl-C is therefore safe: the on-disk set is never partially written and never records a pack that was not verified. The cycle is checkpointed every minute, so an interrupt redoes at most a minute of verification work.Closes #9897