diff --git a/docs/GIT_PROTOCOL_GUIDE.md b/docs/GIT_PROTOCOL_GUIDE.md new file mode 100644 index 00000000..40264aa6 --- /dev/null +++ b/docs/GIT_PROTOCOL_GUIDE.md @@ -0,0 +1,199 @@ +# Git Protocol Abstraction Design and Implementation + +## 1. Project Overview + +The git-internal library implements a transport layer abstraction for the Git smart protocol, separating HTTP and SSH protocol handling from monorepo business code and adapting to any business system through Trait interfaces. + +## 2. Architecture Design + +### 2.1 Layered Architecture + +``` +┌─────────────────────────────────────┐ +│ Business System Integration │ ← Implement Trait interfaces +├─────────────────────────────────────┤ +│ Transport Protocol Adapters │ ← HTTP/SSH handlers +├─────────────────────────────────────┤ +│ Git Smart Protocol Core │ ← Protocol logic implementation +├─────────────────────────────────────┤ +│ Pack File Processing Layer │ ← Object packing/unpacking +└─────────────────────────────────────┘ +``` + +### 2.2 Core Abstraction Interfaces + +**RepositoryAccess Trait** + +- Provides storage layer abstraction, isolating business logic +- Supports reference management, object access, and Pack operations +- Can adapt to any storage backend (filesystem, database, etc.) + +**AuthenticationService Trait** + +- Unified authentication interface supporting HTTP and SSH +- Can integrate with any authentication system (OAuth, JWT, public key, etc.) + +**GitProtocol Core** + +- Transport-agnostic protocol implementation +- Unified info/refs, upload-pack, receive-pack interfaces + +## 3. Implementation Status + +### 3.1 Completed Features + +**Core Protocol** + +- Complete Git smart protocol v1 implementation +- Reference advertisement and capability negotiation +- upload-pack service (clone/fetch operations) +- receive-pack service (push operations) +- Pack file generation and parsing + +**Transport Layer** + +- HTTP transport adapter (request parsing, streaming responses) +- SSH transport adapter (command parsing, authentication integration) +- Transport protocol abstraction (unified interface) + +**Data Processing** + +- Side-band multiplexing +- Progress reporting mechanism +- Object parsing (blob, commit, tree) +- Reference updates and validation + +**Authentication System** + +- HTTP authentication (header-based) +- SSH authentication (public key verification) +- Pluggable authentication architecture + +## 4. Module Organization + +``` +src/protocol/ +├── core.rs # Main abstractions (Trait definitions, GitProtocol) +├── http.rs # HTTP transport adapter +├── ssh.rs # SSH transport adapter +├── smart.rs # Git smart protocol implementation +├── pack.rs # Pack generation and processing +├── types.rs # Protocol types and error definitions +├── utils.rs # Protocol utility functions +└── mod.rs # Module exports +``` + +## 5. HTTP Protocol Abstraction + +### 5.1 Design Features + +- Request path parsing and repository location +- Standard Git HTTP content type handling +- Streaming responses for large repository transfers +- Error mapping to HTTP status codes + +### 5.2 Main Functions + +- `handle_info_refs`: Process reference query requests +- `handle_upload_pack`: Process clone/fetch requests +- `handle_receive_pack`: Process push requests +- `authenticate_http`: HTTP authentication integration + +## 6. SSH Protocol Abstraction + +### 6.1 Design Features + +- Git command line parsing (git-upload-pack, git-receive-pack) +- Repository path extraction and validation +- Direct protocol mapping without HTTP overhead +- Public key authentication integration + +### 6.2 Main Functions + +- Command parsing and dispatching +- Repository path extraction +- Protocol operation mapping +- SSH authentication integration + +## 7. Trait Adaptation Solution + +### 7.1 Storage Adaptation + +Adapt any storage system through RepositoryAccess Trait: + +- Filesystem storage +- Database storage +- Cloud storage services +- Distributed storage + +### 7.2 Authentication Adaptation + +Integrate any authentication system through AuthenticationService Trait: + +- Traditional username/password +- OAuth/JWT tokens +- SSH public key authentication +- Enterprise SSO systems + +### 7.3 Framework Agnostic + +- No dependency on specific web frameworks +- No binding to specific SSH libraries +- No database choice restrictions +- No forced authentication schemes + +## 8. Error Handling and Types + +### 8.1 Protocol Error Types + +- InvalidService: Invalid service request +- RepositoryNotFound: Repository does not exist +- Unauthorized: Authentication failure +- InvalidRequest: Request format error +- Other I/O and internal errors + +### 8.2 Transport Mapping + +Each transport layer is responsible for mapping protocol errors to appropriate transport error formats (HTTP status codes, SSH error messages, etc.). + +## 9. Capabilities and Features + +### 9.1 Supported Git Capabilities + +- side-band-64k: Multiplexed data streams +- ofs-delta: Offset delta objects +- report-status: Push status reporting +- multi_ack_detailed: Detailed acknowledgment negotiation +- no-done: Optimized negotiation flow + +### 9.2 Protocol Features + +- Complete want/have negotiation +- Incremental Pack transmission +- Progress reporting +- Reference update validation + +## 10. Integration Guide + +### 10.1 Implementation Steps + +1. Implement RepositoryAccess Trait to connect storage system +2. Implement AuthenticationService Trait to connect authentication system +3. Create HTTP/SSH handler instances +4. Route requests to appropriate handlers in framework + +### 10.2 Design Principles + +- Separation of concerns: Protocol logic decoupled from business logic +- Interface abstraction: Pluggable architecture through Traits +- Transport agnostic: Same protocol logic supports multiple transports +- Performance focused: Streaming processing, memory efficient + +## 11. Summary + +The git-internal library successfully implements Git protocol transport layer abstraction, separating protocol handling from business logic through clear Trait interfaces. This design supports: + +- **Complete protocol implementation**: Full Git smart protocol v1 functionality +- **Flexible integration solution**: Can adapt to any storage and authentication system +- **Transport layer abstraction**: Unified HTTP and SSH handling +- **High-performance design**: Streaming processing and memory optimization diff --git a/docs/PROTOCOL_ABSTRACTION.md b/docs/PROTOCOL_ABSTRACTION.md deleted file mode 100644 index 2a6f040a..00000000 --- a/docs/PROTOCOL_ABSTRACTION.md +++ /dev/null @@ -1,390 +0,0 @@ -## Git Protocol Abstraction in git-internal - -The git-internal library provides a clean, transport-agnostic abstraction for Git smart protocol operations. This document outlines how the protocol abstraction works and how it separates concerns between transport layers, business logic, and Git protocol handling. - -### 1. Architecture Overview - -The git-internal library implements a layered architecture that cleanly separates different concerns: - - • **Transport Layer**: HTTP and SSH adapters that handle transport-specific details - • **Protocol Layer**: Core Git smart protocol implementation that is transport-agnostic - • **Business Logic Layer**: Trait-based abstractions for repository access and authentication - • **Storage Layer**: Internal Git object handling and pack file operations - -This separation allows the same Git protocol logic to work across different transports and integrate with any storage backend or business system. - -### 2. Core Abstractions - -#### (1) RepositoryAccess Trait - -The `RepositoryAccess` trait provides a clean interface for storage operations without exposing Git protocol details: - -```rust -#[async_trait] -pub trait RepositoryAccess: Send + Sync + Clone { - // Basic repository operations - async fn get_repository_refs(&self, repo_path: &str) -> Result, ProtocolError>; - async fn has_object(&self, repo_path: &str, object_hash: &str) -> Result; - async fn get_object(&self, repo_path: &str, object_hash: &str) -> Result, ProtocolError>; - async fn store_pack_data(&self, repo_path: &str, pack_data: &[u8]) -> Result<(), ProtocolError>; - - // Reference management - async fn update_reference(&self, repo_path: &str, ref_name: &str, old_hash: Option<&str>, new_hash: &str) -> Result<(), ProtocolError>; - - // Pack operations - async fn get_objects_for_pack(&self, repo_path: &str, wants: &[String], haves: &[String]) -> Result, ProtocolError>; - - // Repo-specific utilities - async fn has_default_branch(&self, repo_path: &str) -> Result; - async fn post_receive_hook(&self, repo_path: &str) -> Result<(), ProtocolError>; - - // Optional helpers with default impls (can be overridden): - // - get_blob/get_commit/get_tree - // - commit_exists - // - handle_pack_objects(repo_path, commits, trees, blobs) -} -``` - -This trait abstracts away all storage implementation details, allowing integration with any backend (filesystem, database, cloud storage, etc.). - -#### (2) AuthenticationService Trait - -The `AuthenticationService` trait handles authentication for both HTTP and SSH transports: - -```rust -#[async_trait] -pub trait AuthenticationService: Send + Sync { - /// Authenticate HTTP request using headers - async fn authenticate_http( - &self, - headers: &std::collections::HashMap - ) -> Result<(), ProtocolError>; - - /// Authenticate SSH connection using public key - async fn authenticate_ssh(&self, username: &str, public_key: &[u8]) -> Result<(), ProtocolError>; -} -``` - -This allows for flexible authentication strategies while keeping the protocol layer agnostic to authentication details. - -### 3. Transport Layer Abstraction - -#### (1) HTTP Transport Handler - -The `HttpGitHandler` provides HTTP-specific request handling: - -```rust -pub struct HttpGitHandler { - protocol: GitProtocol, -} - -impl HttpGitHandler { - // Handle HTTP info/refs requests - pub async fn handle_info_refs(&mut self, request_path: &str, query: &str) -> Result<(Vec, &'static str), ProtocolError>; - - // Handle HTTP upload-pack requests (clone/fetch) - pub async fn handle_upload_pack( - &mut self, - request_path: &str, - request_body: &[u8], - ) -> Result< - ( - std::pin::Pin> + Send>>, - &'static str, - ), - ProtocolError, - >; - - // Handle HTTP receive-pack requests (push) - pub async fn handle_receive_pack( - &mut self, - request_path: &str, - request_stream: std::pin::Pin> + Send>>, - ) -> Result< - ( - std::pin::Pin> + Send>>, - &'static str, - ), - ProtocolError, - >; - - // Authenticate HTTP request using headers (call before handle_*) - pub async fn authenticate_http(&self, headers: &std::collections::HashMap) -> Result<(), ProtocolError>; -} -``` - -The HTTP handler includes utility functions for: - -- Extracting repository paths from URLs -- Parsing query parameters -- Setting appropriate content types -- Validating Git requests - -#### (2) SSH Transport Handler - -The `SshGitHandler` provides SSH-specific command handling: - -```rust -pub struct SshGitHandler { - protocol: GitProtocol, -} - -impl SshGitHandler { - // Handle git-upload-pack command - pub async fn handle_upload_pack(&mut self, repo_path: &str, request_data: &[u8]) -> Result> + Send>>, ProtocolError>; - - // Handle git-receive-pack command - pub async fn handle_receive_pack( - &mut self, - repo_path: &str, - request_stream: std::pin::Pin> + Send>>, - ) -> Result> + Send>>, ProtocolError>; - - // Handle info/refs for SSH - pub async fn handle_info_refs(&mut self, repo_path: &str, service: &str) -> Result, ProtocolError>; - - // Authenticate SSH session (call once after handshake) - pub async fn authenticate_ssh(&self, username: &str, public_key: &[u8]) -> Result<(), ProtocolError>; -} -``` - -The SSH handler includes utility functions for: - -- Parsing SSH command lines -- Validating Git SSH commands -- Extracting repository paths from arguments - -### 4. Protocol Layer Implementation - -#### (1) GitProtocol Core - -The `GitProtocol` struct provides the main transport-agnostic interface: - -```rust -pub struct GitProtocol { - smart_protocol: SmartProtocol, -} -``` - -This struct delegates to `SmartProtocol` for the actual Git protocol implementation while providing a clean public API. - -Authentication helpers are also exposed to keep adapters minimal: - -```rust -impl GitProtocol { - pub async fn authenticate_http(&self, headers: &std::collections::HashMap) -> Result<(), ProtocolError>; - pub async fn authenticate_ssh(&self, username: &str, public_key: &[u8]) -> Result<(), ProtocolError>; -} -``` - -#### (2) SmartProtocol Implementation - -The `SmartProtocol` handles all Git smart protocol details: - -- **info/refs**: Advertises repository capabilities and references -- **upload-pack**: Handles fetch/clone operations with pack generation -- **receive-pack**: Handles push operations with pack unpacking - -The protocol implementation is completely transport-agnostic and uses the trait abstractions for all external dependencies. - -### 5. Pack File Handling - -The library includes comprehensive pack file support: - -#### (1) PackGenerator - -Handles pack file generation for upload-pack operations: - -```rust -pub struct PackGenerator<'a, R: RepositoryAccess> { - repo_access: &'a R, -} -``` - -Features: - -- Full pack generation for initial clones -- Incremental pack generation for fetches -- Efficient object traversal and collection -- Streaming pack output - -#### (2) Pack Decoding - -Built on the internal pack module for robust pack file handling: - -- Delta object reconstruction -- Streaming pack decoding -- Memory-efficient processing -- Error recovery and validation - -### 6. Integration Benefits - -#### (1) Transport Independence - -The same Git protocol logic works across: - -- HTTP/HTTPS web servers -- SSH servers -- Custom transport protocols -- Local file system access - -#### (2) Business Logic Separation - -The trait-based design allows: - -- Integration with any storage backend -- Custom authentication strategies -- Flexible repository management -- Easy testing and mocking - -#### (3) Framework Agnostic - -The library doesn't depend on: - -- Specific web frameworks (Axum, Warp, etc.) -- SSH libraries -- Database systems -- Authentication providers - -### 7. Usage Example - -```rust -use git_internal::{GitProtocol, RepositoryAccess, AuthenticationService}; - -// Implement traits for your specific backend -struct MyRepository; -struct MyAuth; - -#[async_trait] -impl RepositoryAccess for MyRepository { - // Implement storage operations... -} - -#[async_trait] -impl AuthenticationService for MyAuth { - // Implement authentication... -} - -// Use with HTTP -let http_handler = HttpGitHandler::new(MyRepository, MyAuth); -// Enforce auth at the entry (framework provides headers) -http_handler.authenticate_http(&headers).await?; -let response = http_handler.handle_info_refs("/repo.git/info/refs", "service=git-upload-pack").await?; - -// Use with SSH -let ssh_handler = SshGitHandler::new(MyRepository, MyAuth); -// Authenticate once after SSH handshake -ssh_handler.authenticate_ssh(username, public_key).await?; -let response = ssh_handler.handle_upload_pack("repo.git", &request_data).await?; - -// Error handling -// On authentication failure, methods return ProtocolError::Unauthorized("...") -``` - -### 8. Advantages - -The abstraction provides significant benefits: - - • **Reduced Coupling**: Business logic is separated from Git protocol details - • **Improved Testability**: Each layer can be tested independently - • **Enhanced Reusability**: The same protocol logic works across different systems - • **Simplified Maintenance**: Changes to transport or storage don't affect protocol logic - • **Performance-Oriented**: Optimized pack handling and streaming operations - -### 9. Design Notes - -- Removed the default `update_ref` method from `RepositoryAccess` to prevent misuse. Callers should always use `update_reference(repo_path, ...)` with an explicit repository path. -- Corrected examples to match the implementation: - - `AuthenticationService::{authenticate_http, authenticate_ssh}` return `Result<(), ProtocolError>`. - - `RepositoryAccess` includes `has_default_branch` and `post_receive_hook` utilities. - - Optional helpers exist with default implementations: `get_blob`, `get_commit`, `get_tree`, `commit_exists`, and `handle_pack_objects(repo_path, ...)`. - -## 10. Capabilities & Feature Matrix - -This section summarizes supported capabilities and how they map to transport handlers: - -- Capability advertisement: `info/refs` exposes negotiated capability lists (`COMMON_CAP_LIST`, `UPLOAD_CAP_LIST`, `RECEIVE_CAP_LIST`). -- LFS advertisement: `COMMON_CAP_LIST` includes `lfs`; negotiation is supported, but object transfer/storage is application-layer (proxy/pass-through only). -- Side-band channels: multiplexed streams supported via `SmartProtocol::build_side_band_format`. -- Pack features: thin-pack, ofs-delta, shallow clones handled by pack encode/decode modules. -- Reference updates: atomic update semantics enforced via `update_reference(repo_path, old, new)`. -- Authentication gate: handlers call `authenticate_http` / `authenticate_ssh` prior to serving operations. - -Supported capability constants: - -```rust -pub const RECEIVE_CAP_LIST: &str = - "report-status report-status-v2 delete-refs quiet atomic no-thin "; -pub const COMMON_CAP_LIST: &str = - "side-band-64k ofs-delta lfs agent=git-internal/0.1.0"; -pub const UPLOAD_CAP_LIST: &str = - "multi_ack_detailed no-done include-tag "; -``` - -Note: trailing spaces in `RECEIVE_CAP_LIST` and `UPLOAD_CAP_LIST` are intentional; they concatenate with `COMMON_CAP_LIST` in `info/refs` advertisement. - -## 11. Error Handling & Mapping - -Protocol errors are framework-agnostic and live under `protocol/types.rs`: - -- Errors represent Git protocol semantics (e.g., `InvalidService`, `RepositoryNotFound`, `InvalidRequest`). -- Storage and parsing failures are wrapped using helpers like `repository_error(...)`. -- Transport adapters are responsible for mapping errors to framework responses. - -Example adapter-side mapping (do not couple the library to HTTP frameworks): - -```rust -// In application layer (e.g. HTTP framework adapter) -fn map_protocol_error_to_http(e: ProtocolError) -> http::StatusCode { - use http::StatusCode; - match e { - ProtocolError::InvalidService(_) => StatusCode::BAD_REQUEST, - ProtocolError::RepositoryNotFound(_) => StatusCode::NOT_FOUND, - ProtocolError::InvalidRequest(_) => StatusCode::BAD_REQUEST, - ProtocolError::Unauthorized(_) => StatusCode::UNAUTHORIZED, - ProtocolError::RepositoryError(_) => StatusCode::INTERNAL_SERVER_ERROR, - } -} -``` - -## 12. Security & Performance Considerations - -- Input validation: strict parsing for service/query/path; reject non-Git endpoints early. -- Resource limits: stream-based pack processing to avoid large memory spikes; consider timeouts and rate limiting at transport layer. -- DoS resilience: cap negotiation and bounded buffering to prevent unbounded allocations. -- Authentication: perform auth before any expensive operations; prefer fail-fast. -- Logging: use `tracing` for structured logs; avoid leaking sensitive data in errors. - -## 13. Testing Strategy & CI - -- Unit tests: cover pkt-line parsing, capability negotiation, ref command parsing. -- Property tests: object parsing and hash invariants using `quickcheck` where applicable. -- Integration tests: pack encode/decode streams on curated datasets (can be long-running). -- CI pipeline: GitHub Actions runs `cargo check`, `cargo clippy -- -D warnings`, and `cargo test`. Long-running tests may be split or marked for scheduled runs if needed. - -## 14. Adoption Guide (Trait-based Integration) - -Steps to integrate with any business system: - -- Implement `RepositoryAccess` for your storage backend: - - Provide refs/object IO, pack storage, ref updates, and optional helpers. -- Implement `AuthenticationService` for your auth strategy: - - Validate HTTP headers or SSH public keys. -- Wire transport adapters: - - HTTP: construct `HttpGitHandler` and route `info/refs`, `upload-pack`, `receive-pack`. - - SSH: construct `SshGitHandler` and dispatch `git-upload-pack` / `git-receive-pack`. -- Keep error mapping in the application layer to stay framework-agnostic. - -Example skeleton: - -```rust -let repo = MyRepoBackend::new(); -let auth = MyAuthService::new(); -let mut http = HttpGitHandler::new(repo.clone(), auth.clone()); -http.authenticate_http(&headers).await?; -let (bytes, content_type) = http.handle_info_refs("/repo.git/info/refs", "service=git-upload-pack").await?; - -let mut ssh = SshGitHandler::new(repo, auth); -ssh.authenticate_ssh(user, key).await?; -let stream = ssh.handle_upload_pack("repo.git", &request).await?; -```