Skip to content

re-add XXH64 to read borg 1.x integrity data, #9935 - #9970

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:re-add-xxh64-9935
Jul 30, 2026
Merged

re-add XXH64 to read borg 1.x integrity data, #9935#9970
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:re-add-xxh64-9935

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

What

Re-adds XXH64 support (removed in #9672 / #9750) so borg transfer can verify the integrity data of a borg 1.x source repository.

borg 1.x wrote XXH64 checksums into the repo index.N / hints.N integrity data. When #9672 switched borg 2.x file integrity to SHA256 and #9750 dropped the xxhash dependency, XXH64 disappeared from SUPPORTED_ALGORITHMS, so reading a legacy repo's integrity data now hits the "Unknown algorithm" path and skips verification. Fixes #9935.

How

Instead of re-introducing the external xxhash PyPI package (and the libxxhash msys2 system dependency that #9750 deliberately removed), this adds a small, dependency-free, streaming XXH64 implementation in Cython to borg.crypto.low_level:

  • cdef class XXH64 (+ a one-shot xxh64() helper) with an interface compatible with the old xxhash.xxh64 FACTORY (XXH64().update().digest()/.hexdigest()), canonical big-endian digest.
  • primes declared as real uint64_t C constants; byte-wise little-endian reads (correct on big-endian hosts too).
  • XXH64FileHashingWrapper re-added and re-registered in SUPPORTED_ALGORITHMS.

XXH64 is only used on the read path — borg 2.x native repos keep writing SHA256. It is non-cryptographic and must not be used as a security mechanism (noted in the code).

Since we implement locally, there is intentionally no change to pyproject.toml or scripts/msys2-install-deps — nothing to re-add.

Tests

  • Official xxHash sanity-check vectors: the test buffer generator is transcribed verbatim from xxHash tests/sanity_test.c; expected digests cross-checked against the reference implementation (empty = ef46db3751d8e999, empty+prime-seed = ac75fda2929b17ef, …). Covers empty / 1-byte / sub-32 / single-stripe / multi-stripe / 4 KiB, seeds 0 and PRIME32, streaming vs one-shot across every chunk alignment, and buffer-protocol inputs.
  • Wrapper tests + a legacy XXH64 integrity round-trip through IntegrityCheckedFile (verify OK + corruption detected).

🤖 Generated with Claude Code

XXH64 (and the XXH64FileHashingWrapper) were removed in borgbackup#9672 / borgbackup#9750,
which switched borg 2.x file integrity to SHA256. But borg 1.x wrote
XXH64 checksums into the repo index/hints integrity data, so we need
XXH64 to verify those files when reading a borg 1.x (legacy) repository
during `borg transfer`.

Rather than re-introducing the external "xxhash" PyPI package (and the
libxxhash system dependency on msys2 that borgbackup#9750 dropped), add a small,
dependency-free, streaming XXH64 implementation in cython to
crypto.low_level. It is only used on the read path; borg 2.x native
repos keep using SHA256. XXH64 is non-cryptographic and must not be used
as a security mechanism.

Tests use the official xxHash sanity-check vectors (the test buffer
generator is transcribed verbatim from xxHash tests/sanity_test.c) and
cover all code paths, streaming vs one-shot, and a legacy XXH64
integrity round-trip through IntegrityCheckedFile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.73%. Comparing base (a50eb6f) to head (15eb490).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9970      +/-   ##
==========================================
+ Coverage   85.70%   85.73%   +0.02%     
==========================================
  Files          95       95              
  Lines       16815    16856      +41     
  Branches     2577     2582       +5     
==========================================
+ Hits        14411    14451      +40     
- Misses       1668     1669       +1     
  Partials      736      736              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann
ThomasWaldmann merged commit 25d24cd into borgbackup:master Jul 30, 2026
31 of 32 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the re-add-xxh64-9935 branch July 30, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

borg2 repo-list|transfer --from-borg1: Unknown algorithm 'XXH64'

1 participant