Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cargo.lock
target/
49 changes: 49 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
[package]
name = "git-internal"
version = "0.1.0"
edition = "2024"
license = "MIT"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
sea-orm = { version = "1.1.12", features = ["sqlx-sqlite"] }
flate2 = { version = "1.1.1", features = ["zlib"] }
serde = { version = "1.0.219", features = ["derive"] }
bstr = "1.12.0"
hex = "0.4.3"
thiserror = "2.0.12"
tracing = "0.1.41"
sha1 = "0.10.6"
colored = "3.0.0"
chrono = { version = "0.4.41", features = ["serde"] }
tracing-subscriber = "0.3.19"
uuid = { version = "1.17.0", features = ["v4"] }
threadpool = "1.8.1"
num_cpus = "1.17.0"
dashmap = "6.1.0"
tokio = { version = "1.45.1", features = ["fs"] }
lru-mem = "0.3.0"
bincode = { version = "2.0.1", features = ["serde"] }
byteorder = "1.5.0"
futures-util = "0.3.31"
bytes = "1.10.1"
axum = { version = "0.8.4", features = ["macros", "json"] }
memchr = "2.7.4"
encoding_rs = "0.8.35"
rayon = "1.10.0"
reqwest = "0.12.18"
ring = "0.17.14"
serde_json = "1.0.140"
ahash = "0.8.12"
diffs = "0.5.1"
libc = "0.2.172"
zstd-sys = { version = "2.0.15+zstd.1.5.7", features = ["experimental"] }

[dev-dependencies]
tokio = { version = "1.45.1", features = ["full"] }
tokio-util = { version = "0.7.15", features = ["io"] }

[features]
default = ["diff_mydrs"]
diff_mydrs = []
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
## Mercury Module - Git Internal Module
## Git Internal Module

Mercury is a high-performance Rust library for encoding and decoding Git internal objects and Pack files. It provides comprehensive support for Git's internal object storage format with advanced features like delta compression, memory management, and concurrent processing.
Git-Internal is a high-performance Rust library for encoding and decoding Git internal objects and Pack files. It provides comprehensive support for Git's internal object storage format with advanced features like delta compression, memory management, and concurrent processing.

## Overview

Mercury is designed to handle Git internal objects and Pack files efficiently, supporting both reading and writing operations with optimized memory usage and multi-threaded processing capabilities. The library implements the complete Git Pack format specification with additional optimizations for large-scale Git operations.
This module is designed to handle Git internal objects and Pack files efficiently, supporting both reading and writing operations with optimized memory usage and multi-threaded processing capabilities. The library implements the complete Git Pack format specification with additional optimizations for large-scale Git operations.

## Key Features

Expand Down Expand Up @@ -74,7 +74,7 @@ Input Stream → Header Parsing → Object Decoding → Delta Resolution → Cac
### Memory Allocator Recommendations

> [!TIP]
> Here are some performance tips that you can use to significantly improve performance when using `Mercury` crates as a dependency.
> Here are some performance tips that you can use to significantly improve performance when using `git-internal` crates as a dependency.

In certain versions of Rust, using `HashMap` on Windows can lead to performance issues. This is due to the allocation strategy of the internal heap memory allocator. To mitigate these performance issues on Windows, you can use [mimalloc](https://github.com/microsoft/mimalloc). (See [this issue](https://github.com/rust-lang/rust/issues/121747) for more details.)

Expand All @@ -87,7 +87,7 @@ A simple approach:
```toml
[target.'cfg(not(windows))'.dependencies]
jemallocator = "0.5.4"

[target.'cfg(windows)'.dependencies]
mimalloc = "0.1.43"
```
Expand All @@ -98,7 +98,7 @@ A simple approach:
#[cfg(not(target_os = "windows"))]
#[global_allocator]
static GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;

#[cfg(target_os = "windows")]
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
Expand All @@ -118,4 +118,4 @@ A simple approach:

### Benchmark

**TODO**
**TODO**
44 changes: 0 additions & 44 deletions mercury/Cargo.toml

This file was deleted.

15 changes: 0 additions & 15 deletions mercury/delta/Cargo.toml

This file was deleted.

33 changes: 0 additions & 33 deletions mercury/delta/README.md

This file was deleted.

27 changes: 0 additions & 27 deletions mercury/zstdelta/Cargo.toml

This file was deleted.

21 changes: 0 additions & 21 deletions mercury/zstdelta/LICENSE

This file was deleted.

3 changes: 0 additions & 3 deletions mercury/zstdelta/README.md

This file was deleted.

46 changes: 0 additions & 46 deletions mercury/zstdelta/src/main.rs

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{errors::GitDeltaError, utils};
use super::{errors::GitDeltaError, utils};
use std::io::{ErrorKind, Read};

const COPY_INSTRUCTION_FLAG: u8 = 1 << 7;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ mod tests {
use std::io::{self, BufReader, Cursor, Read};
use std::path::{Path, PathBuf};

use crate::decode::delta_decode;
use crate::delta::decode::delta_decode;

use super::DeltaDiff;
fn read_zlib_data(path: &Path) -> Result<Vec<u8>, io::Error> {
Expand Down
File renamed without changes.
6 changes: 3 additions & 3 deletions mercury/delta/src/lib.rs → src/delta/mod.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#![allow(dead_code)]

use encode::DeltaDiff;
use rayon::prelude::*;
use std::hash::{DefaultHasher, Hash, Hasher};
Expand All @@ -7,8 +9,6 @@ mod encode;
mod errors;
mod utils;

pub use decode::delta_decode as decode;

const SAMPLE_STEP: usize = 64;
const MIN_DELTA_RATE: f64 = 0.5;

Expand Down Expand Up @@ -108,7 +108,7 @@ pub fn encode(old_data: &[u8], new_data: &[u8]) -> Vec<u8> {

#[cfg(test)]
mod tests {
use crate::{encode_rate, heuristic_encode_rate, heuristic_encode_rate_parallel};
use super::{encode_rate, heuristic_encode_rate, heuristic_encode_rate_parallel};

#[test]
fn test_heuristic_encode_rate() {
Expand Down
File renamed without changes.
16 changes: 0 additions & 16 deletions mercury/src/errors.rs → src/errors.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
use std::string::FromUtf8Error;

use common::errors::MegaError;
use thiserror::Error;

#[derive(Error, Debug)]
Expand Down Expand Up @@ -95,16 +92,3 @@ pub enum GitError {
#[error("{0}")]
CustomError(String),
}

impl From<FromUtf8Error> for GitError {
fn from(err: FromUtf8Error) -> Self {
// convert the FromUtf8Error to GitError and return it
GitError::ConversionError(err.to_string())
}
}

impl From<MegaError> for GitError {
fn from(err: MegaError) -> Self {
GitError::CustomError(err.to_string())
}
}
File renamed without changes.
9 changes: 7 additions & 2 deletions mercury/src/internal/index.rs → src/internal/index.rs
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,8 @@ impl Index {
let mut name = vec![0; name_len];
file.read_exact(&mut name)?;
// The exact encoding is undefined, but the '.' and '/' characters are encoded in 7-bit ASCII
entry.name = String::from_utf8(name)?; // TODO check the encoding
entry.name =
String::from_utf8(name).map_err(|e| GitError::ConversionError(e.to_string()))?; // TODO check the encoding
index
.entries
.insert((entry.name.clone(), entry.flags.stage), entry);
Expand All @@ -288,7 +289,11 @@ impl Index {
while file.bytes_read() + SHA1::SIZE < total_size as usize {
// The remaining 20 bytes must be checksum
let sign = utils::read_bytes(file, 4)?;
println!("{:?}", String::from_utf8(sign.clone())?);
println!(
"{:?}",
String::from_utf8(sign.clone())
.map_err(|e| GitError::ConversionError(e.to_string()))?
);
// If the first byte is 'A'...'Z' the extension is optional and can be ignored.
if sign[0] >= b'A' && sign[0] <= b'Z' {
// Optional extension
Expand Down
File renamed without changes.
File renamed without changes.
Loading