Skip to content

saworbit/orbit

BranchesTags

Repository files navigation

Orbit

Open Resilient Bulk Information Transfer

The intelligent file transfer tool that never gives up.

CI Security Audit codecov Release Rust License GitHub

Project Status: Alpha (v0.6.0)

Orbit is in active development and should be considered alpha-quality software.

  • Safe for: Experimentation, evaluation, non-critical workloads, development environments
  • Use with caution for: Important data transfers (test thoroughly first, maintain backups)
  • Not recommended for: Mission-critical production systems without extensive testing

APIs may change between versions. Some features are experimental and marked as such. See the Feature Maturity Matrix for per-feature stability.

What is Orbit?

Orbit is a file transfer tool built in Rust that combines reliability with performance. Whether you're backing up data, syncing files, transferring to network shares, or moving data to the cloud, Orbit provides intelligent defaults and powerful features.

Philosophy: Intelligence, resilience, and speed.

Why Orbit vs rsync / rclone?

Capability rsync rclone Orbit
Zero-copy transfers No No Yes (platform-specific syscalls, APFS cloning)
Pre-flight safety No No Yes (Disk Guardian: space, permissions, path checks)
Smart auto-tuning No No Yes (Config Optimizer: hardware probing, profile presets)
Content-defined chunking No No Yes (Gear Hash CDC, 99.1% shift resilience)
Global deduplication No No Yes (Universe index, cross-file/backup dedup)
Plain-English previews --dry-run --dry-run orbit explain (human-readable transfer plan)
Interactive setup Manual rclone config orbit init (hardware-aware wizard)
Cloud backends Limited Excellent (40+) S3, Azure, GCS, SSH, SMB (growing)
Resume/checkpoint Partial Partial Chunk-level verification with smart restart
Compression Built-in (zlib) Limited LZ4, Zstd (auto-selection by destination)
Structured audit No No JSONL with HMAC-chained tamper-evident log
Cross-platform Linux/macOS All Linux, macOS, Windows (native optimizations each)
Maturity Battle-tested Battle-tested Alpha (early-stage, actively developed)

Orbit aims to combine rsync/rclone reliability with modern systems programming strengths. rclone remains the best choice for broad cloud provider support; rsync for proven Unix-to-Unix sync. Orbit differentiates on zero-copy performance, safety checks, and intelligent defaults.

Quick Start

# Install a pre-built binary (Linux x86_64, static musl — see /releases for macOS universal, aarch64, Windows)
curl -L https://github.com/saworbit/orbit/releases/download/v0.6.0/orbit-v0.6.0-x86_64-unknown-linux-musl.tar.gz | tar xz
sudo install -m 755 orbit /usr/local/bin/

# …or build from source
git clone https://github.com/saworbit/orbit.git
cd orbit && cargo build --release

# Run the setup wizard
orbit init

# Copy files
orbit /data /backup -R

# Shorthand subcommands
orbit sync /data /backup              # Sync mode
orbit backup /data /backup            # Checksums + Zstd + resume
orbit mirror /data /backup            # Exact replica

# Preview what Orbit would do
orbit explain /data /backup -R --zstd

# Profiles
orbit /data /backup -R --profile fast     # Max speed
orbit /data /backup -R --profile safe     # Max reliability
orbit /data /backup -R --profile network  # Remote-optimized

# Cloud transfers (requires --features network)
orbit /data s3://bucket/path -R
orbit /data ssh://user@host:/path -R

Full Getting Started Guide | CLI Quick Reference

Feature Maturity Matrix

Understanding feature stability helps you make informed decisions about what to use.

Stable Core

Feature Maturity Notes
Core File Copy (Buffered) Stable Well-tested, safe for production use
Zero-Copy Optimization Stable Platform-specific (Linux, macOS, Windows)
Compression (LZ4, Zstd) Stable Reliable for most workloads
Checksum Verification Stable BLAKE3 (default), SHA-256
Local Filesystem Stable Primary use case, thoroughly tested
OrbitSystem Abstraction Stable I/O abstraction layer

Beta (Functional, Needs Real-World Validation)

Feature Maturity Notes
Resume/Checkpoint Beta Works well, needs more edge-case testing
SSH/SFTP Backend Beta Functional, needs more real-world testing
S3 Backend Beta Works well, multipart upload is newer
SMB Backend Beta v0.11.0 upgrade, ready for integration testing
Azure Blob Backend Beta Via object_store crate, added in v0.6.0
GCS Backend Beta Via object_store crate, added in v0.6.0
Delta Detection (V1) Beta rsync-style algorithm
Disk Guardian Beta Pre-flight checks, works well
Config Optimizer Beta Config validation with active probing
Init Wizard Beta Interactive setup with orbit init
Filter System Beta Glob/regex filters
Metadata Preservation Beta Extended attributes are platform-specific
Transfer Shorthands Beta orbit sync, backup, mirror, cp
orbit explain Beta Plain-English transfer plan preview
orbit history Beta Audit log viewer with --json output
Auto-Compression Beta --compress auto picks by destination
Smart Error Suggestions Beta Fuzzy matching, glob detection, path expansion
Manifest System Beta File tracking and verification
Progress/Bandwidth Limiting Beta Integrated across all modes
Audit Logging Beta Structured JSONL telemetry
Global Deduplication (V3) Beta Universe index, O(log N) scalability
Sparse File Handling Beta Zero-chunk detection, hole-aware writes
Hardlink Preservation Beta Inode tracking, --preserve-hardlinks
In-Place Updates Beta Safety tiers, --inplace

Alpha (Experimental -- Expect Changes)

Feature Notes
V2 Architecture (CDC) Content-defined chunking
Semantic Replication Priority-based transfers
Link-Dest++ (Incremental Backup) Chunk-level reference hardlinking
Transfer Journal (Batch Mode) Content-addressed operation journal
Backpressure Dual-threshold flow control
Penalization Exponential backoff deprioritization
Dead-Letter Queue Bounded quarantine for permanent failures
Health Monitor Continuous mid-transfer health checks
Ref-Counted GC WAL-gated garbage collection
Container Packing .orbitpak chunk packing
Typed Provenance Structured event taxonomy
Composable Prioritizers Chainable sort criteria

Alpha features are gated behind feature flags or --smart mode. The stable core works without them.

Key Features

Zero-Copy Performance

Platform-specific syscalls bypass userspace buffers entirely:

  • Linux: copy_file_range (kernel-space copy)
  • macOS: fcopyfile + APFS Copy-on-Write (instant cloning)
  • Windows: CopyFileExW

Disk Guardian (Pre-Flight Safety)

Automatically runs before every transfer to prevent mid-transfer failures:

  • Disk space estimation with configurable safety margins
  • Write permission and filesystem integrity checks
  • Path validation and read-only filesystem detection

Config Optimizer

Validates configuration and auto-tunes settings based on your hardware:

  • CPU, RAM, and I/O throughput probing
  • Conflict detection (e.g., incompatible flag combinations)
  • Hardware probe caching for fast subsequent runs

Smart Error Handling

  • Error classification: Transient errors retry; permanent errors fail fast
  • Exponential backoff with jitter to prevent thundering herd
  • Three modes: Abort (default), Skip (batch-friendly), Partial (resume-friendly)
  • Fuzzy suggestions: Typo detection, glob pattern hints, path expansion

Structured Audit & Telemetry

  • JSONL audit trail with HMAC-SHA256 chaining (tamper-evident)
  • OpenTelemetry integration for distributed tracing
  • orbit history for human-readable audit browsing
  • Machine-readable --json output for all operations

Multi-Protocol Backends

Backend Feature Flag Notes
Local filesystem (default) Zero-copy, all platforms
SSH/SFTP ssh-backend Via ssh2 crate
Amazon S3 s3-native Via object_store (streaming multipart writes); rich orbit s3 ... CLI with presign / versioning / wildcards is opt-in under s3-cli
Azure Blob azure-native Via object_store crate (streaming multipart writes)
Google Cloud Storage gcs-native Via object_store crate (streaming multipart writes)
SMB2/3 smb-native Native pure-Rust

Content-Defined Chunking (V2 -- Alpha)

  • Gear Hash rolling hash with variable-size chunks (8KB-256KB)
  • 99.1% shift resilience for efficient delta transfers
  • BLAKE3 content-addressable chunk IDs
  • Global deduplication via Universe V3 index (redb-backed, O(log N))

V2/V3 features are under active development. Use --smart to opt in.

For detailed feature documentation, see the docs/ directory.

Performance

Local Transfer Benchmarks

File Size Traditional cp Orbit (Zero-Copy) Speedup
10 MB 12 ms 8 ms 1.5x
1 GB 980 ms 340 ms 2.9x
10 GB 9.8 s 3.4 s 2.9x

macOS APFS: File copies complete instantly via Copy-on-Write cloning regardless of file size.

S3 Transfer Performance

  • Streaming multipart writes for objects above 8 MiB, with up to 4 in-flight parts
  • Parallel Workers: Up to 256 concurrent file operations
  • Adaptive Chunking: 5MB-2GB chunks based on file size

Use Cases

# Cloud data lake ingestion
orbit /data/analytics s3://data-lake/raw/2025/ -R --parallel 16 --compress zstd:3

# Enterprise backup with manifest
orbit manifest plan --source /data --dest /backup --output ./manifests
orbit manifest verify --manifest-dir ./manifests

# Hybrid cloud migration
orbit /on-prem/data s3://migration-bucket/data -R --mode sync --resume --parallel 12

# Network share sync
orbit /local/files smb://nas/backup -R --mode sync --resume --retry-attempts 10

Configuration

Configuration File (~/.orbit/orbit.toml)

recursive = true
preserve_metadata = true
resume_enabled = true
verify_checksum = true
compression = { zstd = { level = 5 } }
show_progress = true
retry_attempts = 3
exponential_backoff = true

Run orbit init to generate an optimal config for your hardware.

Configuration Priority

  1. CLI arguments (highest)
  2. --profile preset or auto-network detection
  3. ~/.orbit/orbit.toml (user config)
  4. Built-in defaults (lowest)

Feature Flags (Cargo)

cargo build --release                          # Minimal (~10MB) - local only
cargo build --release --features network       # All network backends via object_store (no aws-sdk-s3)
cargo build --release --features s3-cli        # Adds `orbit s3 ...` subcommands (presign, versioning, wildcards)
cargo build --release --features s3-native,ssh-backend  # Selective

Default build includes only zero-copy. The root binary's optional Tokio features (multi-thread runtime, net, time, etc.) are gated behind network backends and only enabled when those features are turned on. Tokio is still pulled in transitively via the orbit-core-interface and orbit-observability workspace crates regardless of features.

CLI Quick Reference

# Transfer
orbit <SOURCE> <DEST> [FLAGS]
orbit sync|backup|mirror|cp <SOURCE> <DEST> [FLAGS]

# Profiles
--profile fast|safe|backup|network

# S3 operations (build with --features s3-cli)
orbit ls|head|du|rm|mv|mb|rb|cat|pipe|presign s3://...

# Diagnostics
orbit init|doctor|explain|history|stats|capabilities

# Manifests
orbit manifest plan|verify|diff|info

# Batch execution
orbit run --file commands.txt --workers 256

Key flags:

Flag Description
-R / --recursive Copy directories recursively
--resume Resume interrupted transfers
--zstd / --lz4 Compression shorthand
--compress auto Auto-select compression
--workers N Parallel file workers (0=auto)
--checksum blake3 Checksum verification
--dry-run Preview without copying
--json Machine-readable output
--quiet Suppress non-essential output
--verbose Detailed logging

Modular Architecture

Orbit is organized as a Rust workspace with 8 member crates:

Crate Purpose Status
orbit (root) CLI, core engine, transfer orchestration Beta
orbit-core-interface OrbitSystem I/O abstraction Stable
core-manifest Manifest parsing and job orchestration Beta
core-starmap Job planner, Universe V3 index Beta
core-cdc Content-defined chunking (Gear Hash) Alpha
core-semantic Intent-based replication, prioritizers Alpha
core-audit Structured logging, telemetry Beta
orbit-observability Observability and monitoring Beta

See ARCHITECTURE.md for full architecture documentation.

Security

  • Checksum verification: SHA-256, BLAKE3 for data integrity
  • Credential protection: Memory scrubbing on drop, no credential logging
  • No telemetry phone-home: All data stays local
  • Pre-flight validation: Disk Guardian prevents dangerous operations
  • Dependency auditing: deny.toml + CI-integrated cargo deny check
  • S3 encryption: Server-side encryption (AES-256, AWS KMS)
  • Minimal default build: Zero runtime vulnerabilities in default configuration

See SECURITY.md for vulnerability reporting, dependency audit results, and security practices.

Roadmap

Phase 1: Stabilize & Simplify (Current)

  • Stabilize core local + S3/SSH transfers
  • Expand integration test coverage (targeting 80%+ on core paths)
  • Feature-gate experimental V2/V3 behind --smart / alpha flags
  • Continue workspace refactoring and crate extraction

Phase 2: Polish & Differentiate

  • Performance tuning with realistic benchmarks (vs rsync/rclone)
  • Stabilize CDC resume + basic deduplication
  • Enhanced UX (init wizard evolution, progress, telemetry)
  • Grow documentation and examples

Phase 3: Expand & Community

  • Full backend parity across all protocols
  • Plugin framework exploration
  • Encryption at rest
  • Publish reusable crates to crates.io
  • v1.0 when core + advanced features are production-grade

Recently Completed

  • UX overhaul: shorthand subcommands, orbit doctor, output control flags
  • Config resolution hardened: unified transfer path, auto-network detection
  • Saner defaults: preserve_metadata, show_stats, human_readable default to true
  • Simplified CLI: positional arguments, --profile presets, actionable errors
  • Codebase restructuring: main.rs reduced by 45%, subcommands extracted; Cli struct split into 9 grouped Args sub-structs (TransferArgs, ReliabilityArgs, PerformanceArgs, etc.) for cleaner derive, faster parse, and easier maintenance
  • Dependency cleanup: anyhow removed from all first-party crates (unified on thiserror); root binary's heavy Tokio runtime features are feature-gated to network backends, though a minimal tokio is still pulled in transitively via orbit-core-interface and orbit-observability
  • Workspace dependency inheritance: shared dep versions managed centrally

Contributing

Pull requests welcome! See CONTRIBUTING.md for code style, the OrbitSystem pattern, and guidelines.

git clone https://github.com/saworbit/orbit.git
cd orbit
cargo build && cargo test
cargo fmt && cargo clippy

Areas We Need Help

  • Testing on various platforms and real-world scenarios
  • Integration tests for S3 (MinIO), SSH, SMB backends
  • Documentation improvements
  • Bug reports and edge-case discovery
  • Stabilizing beta features toward production readiness

Documentation

Guide Description
Getting Started Installation, setup, and first transfers
S3 User Guide Complete S3 operations guide
GCS User Guide Google Cloud Storage guide
Backend Guide All storage backends
Filter System Glob, regex, and path filters
Init Wizard Guide Deep dive into orbit init
Delta Detection Efficient transfer algorithms
Architecture System design and crate structure
Protocol Guide Protocol-specific details
API Reference cargo doc --open

License

Apache License 2.0 -- See LICENSE for details.

Copyright 2024 Shane Wall
Licensed under the Apache License, Version 2.0

Acknowledgments

  • Built with Rust
  • Inspired by rsync, rclone, and modern transfer tools
  • Thanks to the Rust community for excellent crates

About

Resilient file transfer engine with compression, resume, and parallel operations. Built in Rust.

github.com/saworbit/orbit/blob/main/README.md

Topics

rust cli backup compression zstd file-transfer data-migration parallel-processing checksum-verification resume-support

Resources

Readme

License

Apache-2.0 license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases 5

Orbit v0.6.0 Latest
Dec 10, 2025
+ 4 releases

Sponsor this project

Packages

 
 
 

Contributors

Languages