[mercury] improve slide window algorithm#1407
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull Request Overview
This PR improves the sliding window algorithm for pack encoding in Mercury, focusing on enhancing compression efficiency and rate. The changes introduce a new delta encoding strategy with parallel processing, implement accurate byte counting for decoding, and add comprehensive compression rate tracking in tests.
Key changes:
- Refactored the sliding window algorithm to use magic sorting and parallel processing for better delta compression
- Added
CountingReaderutility for accurate byte tracking during decompression operations - Enhanced delta encoding with heuristic rate calculation functions for improved performance
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| mercury/src/utils.rs | Adds CountingReader wrapper for accurate byte counting during read operations |
| mercury/src/internal/pack/entry.rs | Adds chain_len field to track delta chain length in Entry struct |
| mercury/src/internal/pack/encode.rs | Major refactor of encoding logic with parallel processing and improved delta selection |
| mercury/src/internal/pack/decode.rs | Updates decompression to use CountingReader for accurate input byte tracking |
| mercury/src/internal/pack/cache_object.rs | Initializes new chain_len field in cached objects |
| mercury/delta/src/lib.rs | Adds heuristic delta rate calculation functions for performance optimization |
| mercury/delta/Cargo.toml | Adds rayon dependency for parallel processing |
| mercury/Cargo.toml | Adds ahash dependency for improved hashing performance |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| let is_better = if rate > best_rate + tie_epsilon { | ||
| true | ||
| } else if (rate - best_rate).abs() <= tie_epsilon { | ||
| try_base.0.chain_len > best_base_ref.0.chain_len |
There was a problem hiding this comment.
The condition try_base.0.chain_len > best_base_ref.0.chain_len appears to favor longer chains, but typically shorter delta chains are preferred for performance. This should likely be try_base.0.chain_len < best_base_ref.0.chain_len.
| try_base.0.chain_len > best_base_ref.0.chain_len | |
| try_base.0.chain_len < best_base_ref.0.chain_len |
| let data1 = vec![0u8; 1_000_00]; | ||
| let data2 = vec![1u8; 1_000_00]; |
There was a problem hiding this comment.
The number literal 1_000_00 has an incorrect underscore placement. It should be 1_000_000 or 100_000 depending on the intended size.
| let data1 = vec![0u8; 1_000_00]; | |
| let data2 = vec![1u8; 1_000_00]; | |
| let data1 = vec![0u8; 100_000]; | |
| let data2 = vec![1u8; 100_000]; |
| (a as *const Entry).cmp(&(b as *const Entry)) | ||
| } | ||
|
|
||
| // /// 计算单片段 hash |
There was a problem hiding this comment.
The comment contains Chinese text. Comments should be in English for consistency with the codebase.
| // /// 计算单片段 hash | |
| // /// Calculate hash of a single segment |
|
|
||
| // make some different objects, or decode will fail | ||
| let str_vec = vec!["hello, code,", "hello, world.", "!", "123141251251"]; | ||
| let str_vec = vec!["hello, word", "hello, world.", "!", "123141251251"]; |
There was a problem hiding this comment.
The string 'hello, word' appears to be a typo and should be 'hello, world' to match the pattern of the other test strings.
| let str_vec = vec!["hello, word", "hello, world.", "!", "123141251251"]; | |
| let str_vec = vec!["hello, world", "hello, world.", "!", "123141251251"]; |
| source.push("tests/data/packs/pack-f8bbb573cef7d851957caceb491c073ee8e8de41.pack"); | ||
| // let file_map = crate::test_utils::setup_lfs_file().await; | ||
| // let source = file_map | ||
| // let source = file_mapa |
There was a problem hiding this comment.
The commented variable name 'file_mapa' appears to be a typo and should be 'file_map'.
| // let source = file_mapa | |
| // let source = file_map |
1.improve slide window algorithm
2.enchance compression rate and efficiency
3. add accurate counts bytes fn for decode
Originally, I planned to use zlib-ng to improve decoding efficiency, but it depends on the system’s zlib-ng, which could bring some compatibility issues, so emove it.