init codebase

README.md

@@ -0,0 +1,121 @@
+## Mercury Module - Git Internal Module
+
+Mercury is a high-performance Rust library for encoding and decoding Git internal objects and Pack files. It provides comprehensive support for Git's internal object storage format with advanced features like delta compression, memory management, and concurrent processing.
+
+## Overview
+
+Mercury is designed to handle Git internal objects and Pack files efficiently, supporting both reading and writing operations with optimized memory usage and multi-threaded processing capabilities. The library implements the complete Git Pack format specification with additional optimizations for large-scale Git operations.
+
+## Key Features
+
+### 1. Multi-threaded Processing
+
+- Configurable thread pool for parallel object processing
+- Concurrent delta resolution with dependency management
+- Asynchronous I/O operations for improved performance
+
+### 2. Advanced Memory Management
+
+- LRU-based memory cache with configurable limits
+- Automatic disk spillover for large objects
+- Memory usage tracking and optimization
+- Heap size calculation for accurate memory accounting
+
+### 3. Delta Compression Support
+
+- Offset Delta : References objects by pack file offset
+- Hash Delta : References objects by SHA-1 hash
+- Zstd Delta : Enhanced compression using Zstandard algorithm
+- Intelligent delta chain resolution
+
+### 4. Streaming Support
+
+- Stream-based pack file processing
+- Memory-efficient handling of large pack files
+- Support for network streams and file streams
+
+## Core Algorithms
+
+### Pack Decoding Algorithm
+
+1. Read and validate pack header (PACK signature, version, object count)
+2. For each object in the pack:
+   a. Parse object header (type, size)
+   b. Handle based on object type:
+      - Base objects: Decompress and store directly
+      - Delta objects: Add to waitlist until base is available
+   c. Resolve delta chains when base objects become available
+3. Verify pack checksum
+
+### Delta Resolution Strategy
+
+- Waitlist Management : Delta objects wait for their base objects
+- Dependency Tracking : Maintains offset and hash-based dependency maps
+- Chain Resolution : Recursively applies delta operations
+- Memory Optimization : Calculates expanded object sizes to prevent OOM
+
+### Cache Management
+
+- Two-tier Caching : Memory cache with disk spillover
+- LRU Eviction : Least recently used objects are evicted first
+- Size-based Limits : Configurable memory limits with accurate tracking
+- Async Persistence : Background threads handle disk operations
+
+### Object Processing Pipeline
+
+```
+Input Stream → Header Parsing → Object Decoding → Delta Resolution → Cache Storage → Output
+                     ↓              ↓              ↓              ↓
+                Validation    Decompression   Waitlist Mgmt   Memory Mgmt
+```
+
+## Performance Optimizations
+
+### Memory Allocator Recommendations
+
+> [!TIP]
+> Here are some performance tips that you can use to significantly improve performance when using `Mercury` crates as a dependency.
+
+In certain versions of Rust, using `HashMap` on Windows can lead to performance issues. This is due to the allocation strategy of the internal heap memory allocator. To mitigate these performance issues on Windows, you can use [mimalloc](https://github.com/microsoft/mimalloc). (See [this issue](https://github.com/rust-lang/rust/issues/121747) for more details.)
+
+On other platforms, you can also experiment with [jemalloc](https://github.com/jemalloc/jemalloc) or [mimalloc](https://github.com/microsoft/mimalloc) to potentially improve performance.
+
+A simple approach:
+
+1. Change Cargo.toml to use mimalloc on Windows and jemalloc on other platforms.
+
+   ```toml
+   [target.'cfg(not(windows))'.dependencies]
+   jemallocator = "0.5.4"
+
+   [target.'cfg(windows)'.dependencies]
+   mimalloc = "0.1.43"
+   ```
+
+2. Add `#[global_allocator]` to the main.rs file of the program to specify the allocator.
+
+   ```rust
+   #[cfg(not(target_os = "windows"))]
+   #[global_allocator]
+   static GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;
+
+   #[cfg(target_os = "windows")]
+   #[global_allocator]
+   static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
+   ```
+
+### Concurrent Processing
+
+- Configurable thread pools for CPU-intensive operations
+- Lock-free data structures where possible (DashMap for waitlists)
+- Parallel delta application using Rayon
+
+### 3. I/O Optimization
+
+- Buffered reading with configurable buffer sizes
+- Asynchronous file operations for cache persistence
+- Stream-based processing to minimize memory footprint
+
+### Benchmark
+
+**TODO**

mercury/Cargo.toml

@@ -0,0 +1,44 @@
+[package]
+name = "mercury"
+version = "0.1.0"
+edition = "2024"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+delta = { path = "delta" }
+sapling-zstdelta = { path = "zstdelta" }
+common = { workspace = true }
+callisto = { workspace = true }
+flate2 = { workspace = true, features = ["zlib"] } # enable linking against the libz(C lib); better performance
+serde = { workspace = true, features = ["derive"] }
+bstr = { workspace = true }
+hex = { workspace = true }
+thiserror = { workspace = true }
+tracing = { workspace = true }
+sha1 = { workspace = true }
+colored = { workspace = true }
+chrono = { workspace = true }
+tracing-subscriber = { workspace = true }
+uuid = { workspace = true, features = ["v4"] }
+threadpool = { workspace = true }
+num_cpus = { workspace = true }
+dashmap = { workspace = true }
+tokio = {workspace = true }
+lru-mem = { workspace = true }
+bincode = { workspace = true , features = ["serde"] }
+byteorder = { workspace = true }
+futures-util = { workspace = true }
+bytes = { workspace = true }
+axum = { workspace = true }
+memchr = { workspace = true }
+encoding_rs = { workspace = true }
+rayon = { workspace = true }
+reqwest = { workspace = true }
+ring = { workspace = true }
+serde_json = { workspace = true }
+ahash = { workspace = true }
+
+[dev-dependencies]
+tokio = { workspace = true, features = ["full"] }
+tokio-util = { workspace = true, features = ["io"] }

mercury/delta/Cargo.toml

@@ -0,0 +1,15 @@
+[package]
+name = "delta"
+version = "0.1.0"
+edition = "2024"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+diffs = { workspace = true }
+thiserror = { workspace = true }
+flate2 = { workspace = true, features = ["zlib"] }
+rayon = { workspace = true }
+[features]
+default = ["diff_mydrs"]
+diff_mydrs = []

mercury/delta/README.md

@@ -0,0 +1,33 @@
+# Git Delta
+
+In Git, delta refers to the differences or changes between files or data objects. It is a measure of the amount of change between two versions. By using delta, Git can more efficiently store and transfer changes to files or data objects.
+
+## Example
+
+This module exposes three functions to the outside world:
+
+```rust
+pub fn delta_decode(mut stream : &mut impl Read,base_info: &Vec<u8>) -> Result<Vec<u8>, GitDeltaError>
+
+pub fn delta_encode_rate(old_data: & [u8], new_data: & [u8]) -> f64
+
+pub fn delta_encode(old_data: & [u8], new_data: & [u8]) -> Vec<u8>
+```
+
+If you want to decode a delta data, you need a base data(base_info) and a delta instruction data(stream).
+
+```rust
+use delta;
+
+let delta_result:Result<Vec<u8>, GitDeltaError> = delta::delta_decode(stream, base_info);
+```
+
+On the contrary, if you want to compress another object in delta form based on it, use the encode function
+
+```rust
+use delta;
+
+let delta_data:Vec<u8> = delta::delta_encode(old_data, new_data) ;
+```
+
+In addition, the `delta::delta_encode_rate` function can represent the compression rate of delta

mercury/delta/src/decode/mod.rs

@@ -0,0 +1,73 @@
+use crate::{errors::GitDeltaError, utils};
+use std::io::{ErrorKind, Read};
+
+const COPY_INSTRUCTION_FLAG: u8 = 1 << 7;
+const COPY_OFFSET_BYTES: u8 = 4;
+const COPY_SIZE_BYTES: u8 = 3;
+const COPY_ZERO_SIZE: usize = 0x10000;
+
+pub fn delta_decode(
+    mut stream: &mut impl Read,
+    base_info: &[u8],
+) -> Result<Vec<u8>, GitDeltaError> {
+    // Read the bash object size & Result Size
+    let base_size = utils::read_size_encoding(&mut stream).unwrap();
+    if base_info.len() != base_size {
+        return Err(GitDeltaError::DeltaDecoderError(
+            "base object len is not equal".to_owned(),
+        ));
+    }
+
+    let result_size = utils::read_size_encoding(&mut stream).unwrap();
+    let mut buffer = Vec::with_capacity(result_size);
+    loop {
+        // Check if the stream has ended, meaning the new object is done
+        let instruction = match utils::read_bytes(stream) {
+            Ok([instruction]) => instruction,
+            Err(err) if err.kind() == ErrorKind::UnexpectedEof => break,
+            Err(err) => {
+                panic!("{}", format!("Wrong instruction in delta :{err}"));
+            }
+        };
+
+        if instruction & COPY_INSTRUCTION_FLAG == 0 {
+            // Data instruction; the instruction byte specifies the number of data bytes
+            if instruction == 0 {
+                // Appending 0 bytes doesn't make sense, so git disallows it
+                panic!(
+                    "{}",
+                    GitDeltaError::DeltaDecoderError(String::from("Invalid data instruction"))
+                );
+            }
+
+            // Append the provided bytes
+            let mut data = vec![0; instruction as usize];
+            stream.read_exact(&mut data).unwrap();
+            buffer.extend_from_slice(&data);
+        // result.extend_from_slice(&data);
+        } else {
+            // Copy instruction
+            let mut nonzero_bytes = instruction;
+            let offset =
+                utils::read_partial_int(&mut stream, COPY_OFFSET_BYTES, &mut nonzero_bytes)
+                    .unwrap();
+            let mut size =
+                utils::read_partial_int(&mut stream, COPY_SIZE_BYTES, &mut nonzero_bytes).unwrap();
+            if size == 0 {
+                // Copying 0 bytes doesn't make sense, so git assumes a different size
+                size = COPY_ZERO_SIZE;
+            }
+            // Copy bytes from the base object
+            let base_data = base_info.get(offset..(offset + size)).ok_or_else(|| {
+                GitDeltaError::DeltaDecoderError("Invalid copy instruction".to_string())
+            });
+
+            match base_data {
+                Ok(data) => buffer.extend_from_slice(data),
+                Err(e) => return Err(e),
+            }
+        }
+    }
+    assert!(buffer.len() == result_size);
+    Ok(buffer)
+}