diff --git a/CLAUDE.md b/CLAUDE.md index 12cb618c..95596df5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,29 +61,52 @@ ant-core/src/ ├── lib.rs ├── error.rs # Unified error type (thiserror) ├── config.rs # Platform-appropriate data/log directory paths -└── node/ - ├── mod.rs +├── data/ # Data storage and retrieval +│ ├── mod.rs # Re-exports (Client, DataMap, Wallet, PaymentMode, etc.) +│ ├── error.rs # Data operation errors +│ ├── network.rs # P2P network wrapper (DHT, peer discovery) +│ └── client/ # High-level client API +│ ├── mod.rs # Client, ClientConfig +│ ├── chunk.rs # chunk_put, chunk_get, chunk_exists +│ ├── data.rs # data_upload, data_download, data_map_store/fetch +│ ├── file.rs # file_upload, file_download (streaming self-encryption) +│ ├── payment.rs # pay_for_storage, approve_token_spend +│ ├── quote.rs # get_store_quotes from network peers +│ ├── merkle.rs # Merkle batch payment (PaymentMode enum) +│ └── cache.rs # In-memory LRU chunk cache +└── node/ # Node management + ├── mod.rs # add_nodes, remove_node, reset ├── types.rs # DaemonConfig, DaemonStatus, NodeConfig, NodeInfo, AddNodeOpts, etc. ├── events.rs # NodeEvent enum, EventListener trait ├── binary.rs # Binary resolution (download/cache/validate), ProgressReporter trait ├── registry.rs # Node registry (CRUD, JSON persistence, file locking) + ├── devnet.rs # LocalDevnet (local network + Anvil EVM blockchain) ├── daemon/ │ ├── mod.rs + │ ├── client.rs # Daemon client API (start/stop/status via HTTP) │ ├── server.rs # HTTP server (axum), REST API handlers - │ └── supervisor.rs # Process supervision + │ └── supervisor.rs # Process supervision with backoff └── process/ ├── mod.rs ├── spawn.rs # Spawning node processes └── detach.rs # Platform-specific session detachment ant-cli/src/ -├── main.rs +├── main.rs # Entry point, client/wallet/EVM initialization ├── cli.rs # Top-level clap definition └── commands/ + ├── data/ + │ ├── file.rs # ant file upload/download + │ ├── chunk.rs # ant chunk put/get + │ └── wallet.rs # ant wallet address/balance └── node/ ├── mod.rs ├── add.rs # ant node add command - └── daemon.rs # daemon start/stop/status/info/run commands + ├── daemon.rs # daemon start/stop/status/info/run commands + ├── start.rs # ant node start + ├── stop.rs # ant node stop + ├── status.rs # ant node status + └── reset.rs # ant node reset ``` ## Common Commands diff --git a/README.md b/README.md index ea83bf91..eca8bdec 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,15 @@ # ant — Autonomi Network Client -A unified CLI and library for managing Autonomi network nodes. +A unified CLI and Rust library for storing data on the Autonomi decentralized network and managing Autonomi nodes. ## Overview -This project provides: +This project provides two crates: -- **ant-core** — A headless Rust library containing all business logic for node management. Designed to be consumed by any frontend (CLI, TUI, GUI, AI agents). -- **ant-cli** — A CLI binary (`ant`) built on `ant-core`. +- **ant-core** — A headless Rust library containing all business logic: data storage/retrieval with self-encryption and EVM payments, node lifecycle management, and local devnet tooling. Designed to be consumed by any frontend (CLI, GUI, AI agents, REST clients). +- **ant-cli** — A thin CLI binary (`ant`) built on `ant-core`. -The architecture uses a **daemon-based model**: a single long-running process manages all node processes on the machine, exposing a REST API on localhost. No OS service managers or admin privileges are required. +Data on Autonomi is **content-addressed**. Files are split into encrypted chunks (via [self-encryption](https://en.wikipedia.org/wiki/Convergent_encryption)), each stored at an XOR address derived from its content. A `DataMap` tracks which chunks belong to a file. Payments for storage are made on an EVM-compatible blockchain (Arbitrum). ## Installation @@ -17,11 +17,170 @@ The architecture uses a **daemon-based model**: a single long-running process ma cargo install --path ant-cli ``` -## Commands +## Quick Start -### `ant node daemon` +### Store and retrieve a file (local devnet) -Manage the node management daemon. +```bash +# 1. Start a local devnet (spins up nodes + a local Anvil EVM chain) +# See "Local Development with Devnet" below for details. + +# 2. Upload a file (public — anyone with the address can download) +SECRET_KEY=0x... ant file upload photo.jpg --public \ + --devnet-manifest /tmp/devnet.json --allow-loopback --evm-network local +# Output: ADDRESS=abc123... + +# 3. Download it back +ant file download abc123... -o photo_copy.jpg \ + --devnet-manifest /tmp/devnet.json --allow-loopback --evm-network local +``` + +### Store and retrieve a file (Arbitrum mainnet) + +```bash +# Upload (private — DataMap saved locally) +SECRET_KEY=0x... ant file upload photo.jpg \ + --bootstrap 1.2.3.4:12000 --evm-network arbitrum-one +# Output: DATAMAP_FILE=photo.jpg.datamap + +# Download using the local DataMap +ant file download --datamap photo.jpg.datamap -o photo_copy.jpg \ + --bootstrap 1.2.3.4:12000 --evm-network arbitrum-one +``` + +### Low-level chunk operations + +```bash +# Store a single chunk (< 1 MB) +echo "hello autonomi" | SECRET_KEY=0x... ant chunk put --bootstrap ... +# Output: abc123def456... + +# Retrieve it +ant chunk get abc123def456... --bootstrap ... +# Output: hello autonomi +``` + +--- + +## CLI Reference + +### Global Flags + +| Flag | Description | +|------|-------------| +| `--json` | Output structured JSON instead of human-readable text | +| `--bootstrap ` | Bootstrap peer addresses (comma-separated, for data operations) | +| `--devnet-manifest ` | Path to devnet manifest JSON file | +| `--allow-loopback` | Allow loopback connections (required for local devnet) | +| `--timeout-secs ` | Network operation timeout in seconds (default: 60) | +| `--log-level ` | Log level: trace, debug, info, warn, error (default: info) | +| `--evm-network ` | EVM network for payments: `arbitrum-one`, `arbitrum-sepolia`, or `local` | + +### `ant file` — File Operations + +Upload and download files with automatic chunking, self-encryption, and EVM payment. + +#### `ant file upload ` + +Upload a file to the network. The file is split into encrypted chunks, each paid for via the configured EVM network. Requires `SECRET_KEY` environment variable. + +``` +$ SECRET_KEY=0x... ant file upload my_data.bin --public +ADDRESS=a1b2c3d4e5f6... +MODE=public +CHUNKS=7 +TOTAL_SIZE=450000 +``` + +**Options:** + +| Flag | Description | +|------|-------------| +| `--public` | Store the DataMap on-network (anyone with the address can download). Without this flag, the DataMap is saved to a local `.datamap` file (private). | +| `--merkle` | Force merkle batch payment (single EVM transaction for all chunks). Reduces gas costs for multi-chunk uploads. | +| `--no-merkle` | Disable merkle, always use per-chunk payments. | + +**How it works:** +1. The file is streamed through self-encryption in 8KB reads (never fully loaded into memory). +2. Each encrypted chunk is stored on the network at its XOR content address. +3. Payment is made per-chunk or via a merkle batch transaction (auto-selected by default when >= 64 chunks). +4. A `DataMap` is produced that records which chunks compose the file. +5. In `--public` mode, the DataMap itself is stored as a chunk; the returned address is the DataMap's content address. In private mode, the DataMap is saved to `.datamap` on disk. + +#### `ant file download [ADDRESS]` + +Download a file from the network. + +``` +# Public download (by address) +$ ant file download a1b2c3d4e5f6... -o restored.bin +Downloaded 450000 bytes to restored.bin + +# Private download (from local DataMap) +$ ant file download --datamap my_data.bin.datamap -o restored.bin +Downloaded 450000 bytes to restored.bin +``` + +**Options:** + +| Flag | Description | +|------|-------------| +| `ADDRESS` | Hex-encoded public DataMap address (64 hex chars). | +| `--datamap ` | Path to a local `.datamap` file (for private downloads). | +| `-o, --output ` | Output file path (default: `downloaded_file`). | + +### `ant chunk` — Single-Chunk Operations + +Low-level put/get for individual chunks (max ~1 MB each). Useful for small data or building custom data structures. + +#### `ant chunk put [FILE]` + +Store a single chunk. Reads from `FILE` or stdin. Requires `SECRET_KEY`. + +``` +$ echo "small payload" | SECRET_KEY=0x... ant chunk put +a1b2c3d4... +``` + +#### `ant chunk get
` + +Retrieve a single chunk by its hex-encoded XOR address. + +``` +$ ant chunk get a1b2c3d4... -o output.bin +``` + +**Options:** + +| Flag | Description | +|------|-------------| +| `-o, --output ` | Write to file instead of stdout. | + +### `ant wallet` — Wallet Operations + +Inspect the EVM wallet derived from `SECRET_KEY`. + +#### `ant wallet address` + +Print the wallet's EVM address. + +``` +$ SECRET_KEY=0x... ant wallet address +0x1234567890abcdef... +``` + +#### `ant wallet balance` + +Query the token balance on the configured EVM network. + +``` +$ SECRET_KEY=0x... ant wallet balance --evm-network arbitrum-one +1000000000000000000 +``` + +### `ant node` — Node Management + +Manage Autonomi network nodes via a local daemon process. The daemon runs in the background, exposes a REST API on `127.0.0.1`, and supervises all node processes. #### `ant node daemon start` @@ -32,22 +191,18 @@ $ ant node daemon start Daemon started (pid: 12345, port: 48532) ``` -The daemon runs in the background, listening on a random port on `127.0.0.1`. The port is written to `~/.local/share/ant/daemon.port` for discovery. - #### `ant node daemon stop` -Shut down the running daemon. +Shut down the running daemon. Sends SIGTERM and waits for exit. ``` $ ant node daemon stop Daemon stopped (pid: 12345) ``` -Sends SIGTERM to the daemon process and waits for it to exit. Cleans up the PID and port files. - #### `ant node daemon status` -Show whether the daemon is running and summary statistics. +Show daemon status and node count summary. ``` $ ant node daemon status @@ -61,16 +216,9 @@ Daemon is running Nodes errored: 0 ``` -If the daemon is not running: - -``` -$ ant node daemon status -Daemon is not running -``` - #### `ant node daemon info` -Output connection details for programmatic use. Always outputs JSON regardless of the `--json` flag. +Output connection details as JSON (always JSON, regardless of `--json` flag). AI agents use this to discover the daemon's REST API. ```json { @@ -81,160 +229,368 @@ Output connection details for programmatic use. Always outputs JSON regardless o } ``` -AI agents can use this to bootstrap their connection to the daemon. +#### `ant node add` + +Register one or more nodes in the registry. Does **not** start them. Does **not** require the daemon. + +``` +$ ant node add --rewards-address 0xYourWallet --count 3 --node-port 12000-12002 --path /path/to/antnode +Added 3 node(s): + Node 1: port 12000 + Node 2: port 12001 + Node 3: port 12002 +``` + +If the daemon is running, the command routes through its REST API. Otherwise, it operates directly on the registry file. -### `ant node add` +**Options:** -Register one or more nodes in the node registry. Does **not** start them and does **not** require the daemon to be running. +| Flag | Description | +|------|-------------| +| `--rewards-address ` | Required. EVM wallet address for node earnings. | +| `--count ` | Number of nodes to add (default: 1). | +| `--node-port ` | Port or range (e.g., `12000` or `12000-12004`). | +| `--metrics-port ` | Metrics port or range. | +| `--data-dir-path ` | Custom data directory prefix. | +| `--log-dir-path ` | Custom log directory prefix. | +| `--network-id ` | Network ID (default: 1 for mainnet). | +| `--path ` | Path to a local `antnode` binary. | +| `--version ` | Download a specific version (not yet available). | +| `--url ` | Download binary from a URL archive (not yet available). | +| `--bootstrap ` | Bootstrap peer(s), comma-separated. | +| `--env ` | Environment variables, comma-separated. | + +#### `ant node start` + +Start registered node(s). Requires the daemon to be running. ``` -$ ant node add --rewards-address 0xYourWalletAddress --path /path/to/antnode -Added 1 node(s): - Node 1: - Data dir: ~/.local/share/ant/nodes/node-1 - Log dir: ~/.local/share/ant/nodes/node-1/logs - Port: (none) - Binary: /path/to/antnode - Version: 0.1.0 +$ ant node start # Start all +$ ant node start --service-name node1 # Start specific node ``` -#### Adding multiple nodes with a port range +#### `ant node stop` + +Stop running node(s). Requires the daemon. ``` -$ ant node add --rewards-address 0xYourWallet --count 3 --node-port 12000-12002 --path /path/to/antnode -Added 3 node(s): - Node 1: - Data dir: ~/.local/share/ant/nodes/node-1 - Port: 12000 - Node 2: - Data dir: ~/.local/share/ant/nodes/node-2 - Port: 12001 - Node 3: - Data dir: ~/.local/share/ant/nodes/node-3 - Port: 12002 +$ ant node stop # Stop all +$ ant node stop --service-name node1 # Stop specific node ``` -#### Options +#### `ant node status` -| Flag | Description | -|------|-------------| -| `--rewards-address ` | Required. Wallet address for node earnings | -| `--count ` | Number of nodes to add (default: 1) | -| `--node-port ` | Port or port range (e.g., `12000` or `12000-12004`) | -| `--metrics-port ` | Metrics port or range | -| `--data-dir-path ` | Custom data directory prefix | -| `--log-dir-path ` | Custom log directory prefix | -| `--network-id ` | Network ID (default: 1 for mainnet) | -| `--path ` | Path to a local node binary | -| `--version ` | Download a specific version (not yet implemented) | -| `--url ` | Download binary from a URL (not yet implemented) | -| `--bootstrap ` | Bootstrap peer(s), comma-separated | -| `--env ` | Environment variables, comma-separated | +Display all nodes and their current status. -If the daemon is running, the command routes through its REST API. Otherwise, it operates directly on the registry file. +#### `ant node reset` -### `ant node start` +Remove all node data, log directories, and clear the registry. All nodes must be stopped first. -Start registered node(s). Requires the daemon to be running. With no arguments, starts all registered nodes. Use `--service-name` to start a specific node. +``` +$ ant node reset --force +``` +--- + +## REST API + +When the daemon is running, it exposes a REST API on `127.0.0.1:`. Discover the port via `ant node daemon info` or by reading `~/.local/share/ant/daemon.port`. + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/v1/status` | Daemon health, uptime, node count summary | +| GET | `/api/v1/events` | SSE stream of real-time node events | +| GET | `/api/v1/nodes/status` | Node status summary | +| POST | `/api/v1/nodes` | Add nodes to the registry | +| DELETE | `/api/v1/nodes/{id}` | Remove a node | +| POST | `/api/v1/nodes/{id}/start` | Start a specific node | +| POST | `/api/v1/nodes/start-all` | Start all registered nodes | +| POST | `/api/v1/nodes/{id}/stop` | Stop a specific node | +| POST | `/api/v1/nodes/stop-all` | Stop all running nodes | +| POST | `/api/v1/reset` | Reset all node state (fails if nodes running) | +| GET | `/api/v1/openapi.json` | OpenAPI 3.1 specification | +| GET | `/console` | Web status console (HTML) | + +### Error Envelope + +All error responses use a consistent envelope: + +```json +{ + "error": { + "code": "NODE_NOT_FOUND", + "message": "No node with id 42" + } +} ``` -$ ant node start -Started 3 node(s): - node1 (1) — PID 12345 - node2 (2) — PID 12346 - node3 (3) — PID 12347 + +### Idempotency + +409 Conflict responses include `current_state` so retrying clients can confirm the desired state already exists: + +```json +{ + "error": { + "code": "NODE_ALREADY_RUNNING", + "message": "Node 3 is already running" + }, + "current_state": { + "node_id": 3, + "status": "running", + "pid": 12345, + "uptime_secs": 3600 + } +} ``` -#### Starting a specific node +### SSE Events + +`GET /api/v1/events` streams real-time node lifecycle events: ``` -$ ant node start --service-name node1 -Node node1 (1) started (PID 12345) +event: node_started +data: {"node_id": 1, "pid": 12345} + +event: node_crashed +data: {"node_id": 2, "exit_code": 1} ``` -If the daemon is not running, the command will fail with an error asking you to start it first. +Event types: `node_starting`, `node_started`, `node_stopping`, `node_stopped`, `node_crashed`, `node_restarting`, `node_errored`, `download_started`, `download_progress`, `download_complete`. -#### Options +--- -| Flag | Description | -|------|-------------| -| `--service-name ` | Start a specific node by its service name (e.g., `node1`) | +## Rust Library API (`ant-core`) + +The `ant-core` crate exposes the full API programmatically. Add it as a dependency: -### `ant node stop` +```toml +[dependencies] +ant-core = { path = "ant-core" } +``` + +### Connecting to the Network + +```rust +use ant_core::data::{Client, ClientConfig}; -Stop running node(s). Requires the daemon to be running. With no arguments, stops all running nodes. Use `--service-name` to stop a specific node. +// Connect to bootstrap peers +let client = Client::connect(&["1.2.3.4:12000".parse()?], ClientConfig::default()).await?; +// Attach a wallet for paid operations (uploads) +use ant_core::data::{Wallet, EvmNetwork}; +let wallet = Wallet::new_from_private_key(EvmNetwork::ArbitrumOne, "0xprivate_key...")?; +let client = client.with_wallet(wallet); +client.approve_token_spend().await?; ``` -$ ant node stop -Stopped 3 node(s): - node1 (1) - node2 (2) - node3 (3) + +### Uploading and Downloading Files + +```rust +use std::path::Path; +use ant_core::data::PaymentMode; + +// Upload a file (streamed, never fully loaded into memory) +let result = client.file_upload(Path::new("photo.jpg")).await?; +println!("Stored {} chunks", result.chunks_stored); + +// Upload with explicit payment mode +let result = client.file_upload_with_mode(Path::new("photo.jpg"), PaymentMode::Merkle).await?; + +// Store DataMap publicly (anyone with the address can download) +let public_address = client.data_map_store(&result.data_map).await?; + +// Download a public file +let data_map = client.data_map_fetch(&public_address).await?; +client.file_download(&data_map, Path::new("photo_copy.jpg")).await?; ``` -#### Stopping a specific node +### Uploading and Downloading In-Memory Data +```rust +use bytes::Bytes; + +// Upload bytes (encrypted + chunked automatically) +let result = client.data_upload(Bytes::from("hello autonomi")).await?; + +// Download and decrypt +let content = client.data_download(&result.data_map).await?; +assert_eq!(content, Bytes::from("hello autonomi")); ``` -$ ant node stop --service-name node1 -Node node1 (1) stopped + +### Low-Level Chunk Operations + +```rust +use ant_core::data::XorName; + +// Store a single chunk (< MAX_CHUNK_SIZE bytes) +let address: XorName = client.chunk_put(Bytes::from("small data")).await?; + +// Retrieve it +if let Some(chunk) = client.chunk_get(&address).await? { + println!("Got {} bytes", chunk.content.len()); +} + +// Check existence without downloading +let exists: bool = client.chunk_exists(&address).await?; ``` -If the daemon is not running, the command will fail with an error asking you to start it first. +### Payment Modes -#### Options +Autonomi supports two payment strategies: -| Flag | Description | +| Mode | Description | |------|-------------| -| `--service-name ` | Stop a specific node by its service name (e.g., `node1`) | +| `PaymentMode::Single` | One EVM transaction per chunk. Simple but more gas for many chunks. | +| `PaymentMode::Merkle` | Single EVM transaction for a batch of chunks via merkle proof. Lower gas for large uploads. | +| `PaymentMode::Auto` | Default. Uses merkle when chunk count >= 64, otherwise per-chunk. | -### `ant node reset` +### Local Development with Devnet -Remove all node data directories, log directories, and clear the registry, returning the system to a clean state. All nodes must be stopped before running this command. +`LocalDevnet` spins up a local Autonomi network with an embedded Anvil EVM blockchain for development and testing: +```rust +use ant_core::data::LocalDevnet; + +// Start a minimal devnet (5 nodes + Anvil EVM chain) +let mut devnet = LocalDevnet::start_minimal().await?; + +// Create a client with a pre-funded wallet (ready to upload) +let client = devnet.create_funded_client().await?; + +// Upload data +let result = client.data_upload(Bytes::from("test payload")).await?; + +// Write manifest for CLI usage +devnet.write_manifest(Path::new("/tmp/devnet.json")).await?; + +// Clean up +devnet.shutdown().await?; ``` -$ ant node reset -This will remove all node data directories, log directories, and clear the registry. -Are you sure? [y/N] y -Reset complete: - Nodes cleared: 3 - Removed data dir: ~/.local/share/ant/nodes/node-1 - Removed data dir: ~/.local/share/ant/nodes/node-2 - Removed data dir: ~/.local/share/ant/nodes/node-3 - Removed log dir: ~/.local/share/ant/logs/node-1/logs - Removed log dir: ~/.local/share/ant/logs/node-2/logs - Removed log dir: ~/.local/share/ant/logs/node-3/logs + +The manifest JSON can be passed to the CLI via `--devnet-manifest` for local testing. + +### Chunk Cache + +The client includes an in-memory LRU cache for recently accessed chunks: + +```rust +let cache = client.chunk_cache(); +cache.put(address, content); +if let Some(data) = cache.get(&address) { /* ... */ } ``` -#### Options +### Node Management (Programmatic) -| Flag | Description | -|------|-------------| -| `--force` | Skip the confirmation prompt | +```rust +use ant_core::node::{add_nodes, AddNodeOpts, BinarySource}; +use ant_core::node::binary::NoopProgress; +use ant_core::config::data_dir; -If any nodes are running, the command will fail with an error asking you to stop them first. If the daemon is running, the command routes through its REST API. +let opts = AddNodeOpts { + count: 3, + rewards_address: "0xYourWallet".to_string(), + binary_source: BinarySource::LocalPath("/path/to/antnode".into()), + ..Default::default() +}; -### Global Flags +let result = add_nodes(opts, &data_dir()?.join("node_registry.json"), &NoopProgress).await?; +``` -| Flag | Description | -|------|-------------| -| `--json` | Output structured JSON instead of human-readable text | +--- -## REST API +## Architecture -When the daemon is running, it exposes these endpoints on `127.0.0.1:`: +``` +┌──────────┐ HTTP ┌──────────────────────────────────┐ +│ ant CLI │──────────────▶│ ant daemon │ +└──────────┘ 127.0.0.1 │ │ + │ ┌────────────┐ ┌────────────┐ │ +┌──────────┐ HTTP │ │ antnode 1 │ │ antnode 2 │ │ +│ Web UI │──────────────▶│ └────────────┘ └────────────┘ │ +└──────────┘ │ ┌────────────┐ ┌────────────┐ │ + │ │ antnode 3 │ │ antnode N │ │ +┌──────────┐ HTTP │ └────────────┘ └────────────┘ │ +│ AI Agent │──────────────▶│ │ +└──────────┘ └──────────────────────────────────┘ + │ + ▼ + node_registry.json +``` -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/v1/status` | Daemon health, uptime, node count summary | -| GET | `/api/v1/events` | SSE stream of real-time node events | -| POST | `/api/v1/nodes` | Add one or more nodes to the registry | -| DELETE | `/api/v1/nodes/{id}` | Remove a node from the registry | -| POST | `/api/v1/nodes/{id}/start` | Start a specific node | -| POST | `/api/v1/nodes/start-all` | Start all registered nodes | -| POST | `/api/v1/nodes/{id}/stop` | Stop a specific node | -| POST | `/api/v1/nodes/stop-all` | Stop all running nodes | -| POST | `/api/v1/reset` | Reset all node state (fails if nodes running) | -| GET | `/api/v1/openapi.json` | OpenAPI 3.1 specification | +The daemon manages node processes, exposing a REST API on localhost. No admin privileges required. The CLI, web UI, and AI agents all communicate over HTTP. + +Data operations (upload/download) go directly to the P2P network — they do not require the daemon. The daemon is only needed for node management. + +## Project Structure + +``` +├── ant-core/ # Headless library — all business logic +│ ├── src/ +│ │ ├── lib.rs +│ │ ├── config.rs # Platform-appropriate data/log paths +│ │ ├── error.rs # Unified error type +│ │ ├── data/ # Data storage and retrieval +│ │ │ ├── mod.rs # Re-exports and module declarations +│ │ │ ├── error.rs # Data operation errors +│ │ │ ├── network.rs # P2P network wrapper +│ │ │ └── client/ # High-level client API +│ │ │ ├── mod.rs # Client, ClientConfig +│ │ │ ├── chunk.rs # chunk_put, chunk_get, chunk_exists +│ │ │ ├── data.rs # data_upload, data_download, data_map_store/fetch +│ │ │ ├── file.rs # file_upload, file_download (streaming) +│ │ │ ├── payment.rs # pay_for_storage, approve_token_spend +│ │ │ ├── quote.rs # get_store_quotes from peers +│ │ │ ├── merkle.rs # Merkle batch payment (PaymentMode) +│ │ │ └── cache.rs # In-memory LRU chunk cache +│ │ └── node/ # Node management +│ │ ├── mod.rs # add_nodes, remove_node, reset +│ │ ├── types.rs # DaemonConfig, NodeConfig, AddNodeOpts, etc. +│ │ ├── events.rs # NodeEvent enum, EventListener trait +│ │ ├── binary.rs # Binary resolution, ProgressReporter trait +│ │ ├── registry.rs # Node registry (CRUD, JSON, file locking) +│ │ ├── devnet.rs # LocalDevnet (local network + Anvil EVM) +│ │ ├── daemon/ +│ │ │ ├── mod.rs +│ │ │ ├── client.rs # Daemon client (start/stop/status via HTTP) +│ │ │ ├── server.rs # HTTP server (axum), REST API handlers +│ │ │ └── supervisor.rs # Process supervision with backoff +│ │ └── process/ +│ │ ├── mod.rs +│ │ ├── spawn.rs # Spawning node processes +│ │ └── detach.rs # Platform-specific session detachment +│ └── tests/ # Integration tests +├── ant-cli/ # CLI binary (thin adapter layer) +│ └── src/ +│ ├── main.rs # Entry point, client/wallet initialization +│ ├── cli.rs # clap argument definitions +│ └── commands/ +│ ├── data/ +│ │ ├── file.rs # ant file upload/download +│ │ ├── chunk.rs # ant chunk put/get +│ │ └── wallet.rs # ant wallet address/balance +│ └── node/ +│ ├── add.rs # ant node add +│ ├── daemon.rs # ant node daemon start/stop/status/info/run +│ ├── start.rs # ant node start +│ ├── stop.rs # ant node stop +│ ├── status.rs # ant node status +│ └── reset.rs # ant node reset +└── docs/ # Architecture documentation +``` + +## EVM Networks + +| Value | Network | Use case | +|-------|---------|----------| +| `arbitrum-one` | Arbitrum mainnet | Production | +| `arbitrum-sepolia` | Arbitrum Sepolia testnet | Staging / testing | +| `local` | Custom (Anvil) | Local development (requires `--devnet-manifest`) | + +## Environment Variables + +| Variable | Required for | Description | +|----------|-------------|-------------| +| `SECRET_KEY` | Uploads, wallet commands | EVM private key (hex, with or without `0x` prefix) | ## Development @@ -242,35 +598,21 @@ When the daemon is running, it exposes these endpoints on `127.0.0.1:`: # Build cargo build -# Run tests +# Run all tests cargo test --all # Lint cargo clippy --all-targets --all-features -- -D warnings -# Format -cargo fmt --all +# Format check +cargo fmt --all -- --check # Run the CLI +cargo run --bin ant -- --help +cargo run --bin ant -- file upload photo.jpg --public --devnet-manifest /tmp/devnet.json --allow-loopback cargo run --bin ant -- node daemon status ``` -## Project Structure - -``` -├── ant-core/ # Headless library — all business logic -│ ├── src/ -│ │ ├── config.rs # Platform-appropriate paths -│ │ ├── error.rs # Unified error type -│ │ └── node/ # Node management modules -│ └── tests/ # Integration tests -├── ant-cli/ # CLI binary (thin adapter layer) -│ └── src/ -│ ├── cli.rs # clap argument definitions -│ └── commands/ # Command handlers -└── docs/ # Architecture documentation -``` - ## License TBD diff --git a/docs/data-storage-architecture.md b/docs/data-storage-architecture.md new file mode 100644 index 00000000..bc267537 --- /dev/null +++ b/docs/data-storage-architecture.md @@ -0,0 +1,246 @@ +# Architecture: Data Storage and Retrieval + +## Overview + +Autonomi stores data as content-addressed, encrypted chunks on a P2P network. All data — files, +blobs, metadata — is broken into chunks via [self-encryption](https://en.wikipedia.org/wiki/Convergent_encryption), +each stored at an XOR address derived from its content. Storage is paid for via EVM transactions +on an Arbitrum-compatible blockchain. + +The `ant_core::data` module provides the complete API. No daemon is needed for data operations — +the client connects directly to the P2P network. + +## Key Concepts + +### Content Addressing (XorName) + +Every chunk is stored at a 32-byte `XorName` derived from its content hash. This means: +- Identical data always maps to the same address (deduplication is automatic). +- You cannot modify data in place — you store a new chunk at a new address. +- Addresses are deterministic: you can compute a chunk's address before storing it. + +### Self-Encryption + +Self-encryption (convergent encryption) encrypts data using keys derived from the data itself: +1. Data is split into chunks. +2. Each chunk is encrypted using keys derived from neighboring chunks. +3. The encryption is deterministic — the same input always produces the same encrypted chunks. +4. A `DataMap` records all chunk addresses and encryption parameters needed for reconstruction. + +The `DataMap` is the "key" to your data. Whoever holds the DataMap can reconstruct the original +content. DataMaps can be: +- **Kept private** (saved to a local file) — only the holder can download. +- **Stored publicly** (uploaded as a chunk) — anyone with the chunk address can download. + +### Payment Model + +Each chunk stored on the network requires a one-time payment to the nodes that store it: + +1. **Quote collection**: Client asks close-group peers for storage quotes (price + terms). +2. **Payment**: Client sends an EVM transaction paying the quoted amount. +3. **Storage proof**: The payment transaction produces a proof that the client presents when + uploading the chunk. +4. **Verification**: Storing nodes verify the proof before accepting the chunk. + +Two payment modes are available: +- **Single** — One EVM transaction per chunk. Simple but costly for many chunks. +- **Merkle** — A single EVM transaction covers a batch of chunks via a merkle tree proof. This + reduces gas costs significantly for multi-chunk uploads (e.g., files). +- **Auto** (default) — Uses merkle when >= 64 chunks, otherwise per-chunk. + +## Client API + +### Creating a Client + +```rust +use ant_core::data::{Client, ClientConfig}; + +// Connect to known bootstrap peers +let client = Client::connect( + &["1.2.3.4:12000".parse()?], + ClientConfig::default(), +).await?; +``` + +`ClientConfig` controls: +- `timeout_secs` — Timeout for each network operation (default: 10s). +- `close_group_size` — Number of closest peers to query for routing (default: protocol constant). + +### Attaching a Wallet + +Upload operations require a funded EVM wallet: + +```rust +use ant_core::data::{Wallet, EvmNetwork}; + +let wallet = Wallet::new_from_private_key( + EvmNetwork::ArbitrumOne, + "your_private_key_hex", +)?; +let client = client.with_wallet(wallet); + +// Approve the payment contract to spend tokens (one-time per session) +client.approve_token_spend().await?; +``` + +### File Operations + +Files are the primary high-level API. They handle chunking, encryption, payment, and storage +automatically. + +```rust +use std::path::Path; +use ant_core::data::PaymentMode; + +// Upload (streaming — file never fully loaded into memory) +let result = client.file_upload(Path::new("video.mp4")).await?; +// result.data_map — the DataMap (save this to download later) +// result.chunks_stored — number of chunks uploaded + +// Upload with explicit payment mode +let result = client.file_upload_with_mode( + Path::new("large_dataset.tar"), + PaymentMode::Merkle, +).await?; + +// Make data publicly downloadable +let public_address: [u8; 32] = client.data_map_store(&result.data_map).await?; + +// Download a file +let data_map = client.data_map_fetch(&public_address).await?; +let bytes_written = client.file_download(&data_map, Path::new("output.mp4")).await?; +``` + +### In-Memory Data Operations + +For data already in memory (API payloads, generated content, etc.): + +```rust +use bytes::Bytes; + +// Upload bytes +let result = client.data_upload(Bytes::from(my_data)).await?; + +// Upload with payment mode control +let result = client.data_upload_with_mode( + Bytes::from(my_data), + PaymentMode::Merkle, +).await?; + +// Download and decrypt +let content: Bytes = client.data_download(&result.data_map).await?; +``` + +### Chunk Operations + +Low-level operations on individual chunks (each up to `MAX_CHUNK_SIZE` bytes): + +```rust +use ant_core::data::XorName; + +// Store a chunk (pays for storage automatically) +let address: XorName = client.chunk_put(Bytes::from("small data")).await?; + +// Retrieve a chunk +if let Some(chunk) = client.chunk_get(&address).await? { + println!("Content: {} bytes", chunk.content.len()); +} + +// Check existence without downloading +let exists: bool = client.chunk_exists(&address).await?; +``` + +### Chunk Cache + +The client maintains an in-memory LRU cache (default capacity: 1024 chunks) to avoid redundant +network fetches: + +```rust +let cache = client.chunk_cache(); +cache.contains(&address); // Check without fetch +cache.len(); // Current cache size +cache.clear(); // Evict all +``` + +The cache is used transparently by `chunk_get` — repeated reads of the same address hit the cache. + +## Network Layer + +The `Network` struct wraps a P2P node and provides DHT operations: + +```rust +use ant_core::data::Network; + +let network = client.network(); + +// Find peers closest to a target address (XOR distance) +let peers = network.find_closest_peers(&target_hash, 8).await?; + +// List currently connected peers +let connected = network.connected_peers().await; + +// Get local peer ID +let my_id = network.peer_id(); +``` + +## Local Devnet + +`LocalDevnet` provides a complete local development environment: + +```rust +use ant_core::data::LocalDevnet; + +// Start 5 Autonomi nodes + a local Anvil EVM blockchain +let mut devnet = LocalDevnet::start_minimal().await?; + +// Get a funded client (wallet with test tokens, pre-approved) +let client = devnet.create_funded_client().await?; + +// Use the client normally +let result = client.data_upload(Bytes::from("test")).await?; +let content = client.data_download(&result.data_map).await?; + +// Export manifest for CLI usage: +// ant file upload ... --devnet-manifest /tmp/devnet.json --allow-loopback --evm-network local +devnet.write_manifest(Path::new("/tmp/devnet.json")).await?; + +// Shut down everything +devnet.shutdown().await?; +``` + +Devnet configurations: +- `LocalDevnet::start_minimal()` — 5 nodes (fast startup, good for unit tests). +- `LocalDevnet::start_small()` — 10 nodes (better for integration tests). +- `LocalDevnet::start(config)` — Custom `DevnetConfig` with arbitrary node count, ports, etc. + +## Error Handling + +All data operations return `ant_core::data::Result`, where `Error` covers: + +| Variant | Meaning | +|---------|---------| +| `Network` | P2P network failure (connection, routing) | +| `Storage` | Storage node rejected the operation | +| `Payment` | EVM transaction failed or insufficient funds | +| `Protocol` | Protocol-level error (bad messages, remote errors) | +| `Timeout` | Peer didn't respond within the timeout | +| `InsufficientPeers` | Not enough peers found for the operation | +| `Encryption` | Self-encryption or decryption failure | +| `InvalidData` | Data integrity check failed | +| `AlreadyStored` | Chunk already exists on the network (not an error in practice) | +| `Io` | Local filesystem error | +| `Serialization` | DataMap or protocol message serialization failure | +| `Crypto` | Cryptographic operation failed | +| `Config` | Configuration error | +| `SignatureVerification` | BLS/ML-DSA signature verification failed | + +## EVM Networks + +| Network | Chain | Use case | +|---------|-------|----------| +| `EvmNetwork::ArbitrumOne` | Arbitrum mainnet | Production | +| `EvmNetwork::ArbitrumSepoliaTest` | Arbitrum Sepolia | Staging / testing | +| `EvmNetwork::Custom(...)` | Any EVM chain | Local dev (Anvil), custom deployments | + +For local development, `LocalDevnet` handles all EVM setup automatically. For custom deployments, +construct a `CustomNetwork` with the RPC URL, token address, and payment contract addresses. diff --git a/docs/node-management-architecture.md b/docs/node-management-architecture.md index e48314de..f64e3a15 100644 --- a/docs/node-management-architecture.md +++ b/docs/node-management-architecture.md @@ -62,19 +62,33 @@ All business logic. No UI code, no terminal output. ``` ant-core/src/ ├── lib.rs -├── error.rs # Unified error type -├── config.rs # Shared configuration types -└── node/ - ├── mod.rs - ├── types.rs # NodeInfo, NodeStatus, AddNodeOpts, etc. +├── config.rs # Platform-appropriate data/log directory paths +├── error.rs # Unified error type (thiserror) +├── data/ # Data storage and retrieval +│ ├── mod.rs # Re-exports and module declarations +│ ├── error.rs # Data operation errors +│ ├── network.rs # P2P network wrapper (DHT, peer discovery) +│ └── client/ # High-level client API +│ ├── mod.rs # Client, ClientConfig +│ ├── chunk.rs # chunk_put, chunk_get, chunk_exists +│ ├── data.rs # data_upload, data_download, data_map_store/fetch +│ ├── file.rs # file_upload, file_download (streaming) +│ ├── payment.rs # pay_for_storage, approve_token_spend +│ ├── quote.rs # get_store_quotes from network peers +│ ├── merkle.rs # Merkle batch payment (PaymentMode enum) +│ └── cache.rs # In-memory LRU chunk cache +└── node/ # Node management + ├── mod.rs # add_nodes, remove_node, reset + ├── types.rs # DaemonConfig, NodeConfig, AddNodeOpts, etc. ├── events.rs # NodeEvent enum, EventListener trait - ├── progress.rs # ProgressReporter trait - ├── registry.rs # Node registry (CRUD, JSON persistence) + ├── binary.rs # Binary resolution, ProgressReporter trait + ├── registry.rs # Node registry (CRUD, JSON persistence, file locking) + ├── devnet.rs # LocalDevnet (local network + Anvil EVM) ├── daemon/ │ ├── mod.rs + │ ├── client.rs # Daemon client API (start/stop/status via HTTP) │ ├── server.rs # HTTP server (axum), REST API handlers - │ ├── supervisor.rs # Process supervision loop - │ └── console.rs # Static web UI serving + │ └── supervisor.rs # Process supervision loop └── process/ ├── mod.rs ├── spawn.rs # Spawning node processes @@ -90,18 +104,21 @@ Thin adapter layer. Parses arguments, calls `ant-core`, formats output. ``` ant-cli/src/ -├── main.rs +├── main.rs # Entry point, client/wallet/EVM initialization ├── cli.rs # Top-level clap definition └── commands/ + ├── data/ + │ ├── file.rs # ant file upload/download + │ ├── chunk.rs # ant chunk put/get + │ └── wallet.rs # ant wallet address/balance └── node/ ├── mod.rs - ├── add.rs # Calls ant_core::node::registry directly - ├── start.rs # Sends HTTP to daemon - ├── stop.rs # Sends HTTP to daemon - ├── remove.rs # Calls ant_core::node::registry directly - ├── list.rs # Calls registry or daemon - ├── status.rs # Sends HTTP to daemon - └── daemon.rs # Launches/stops/queries daemon process + ├── add.rs # ant node add + ├── daemon.rs # ant node daemon start/stop/status/info/run + ├── start.rs # ant node start + ├── stop.rs # ant node stop + ├── status.rs # ant node status + └── reset.rs # ant node reset ``` When the daemon is not running, the CLI calls `ant-core` directly for registry operations (`add`, @@ -112,24 +129,83 @@ needing the CLI. ## Command Structure ``` -ant [--json] # Global flag: output structured JSON instead of human text -ant node -├── daemon start # Launch daemon (detached) -├── daemon stop # Shut down daemon -├── daemon status # Is daemon running, summary stats -├── daemon info # Connection details (port, PID) for programmatic use -├── add [options] # Register node(s) in the registry -├── remove # Remove node from registry -├── start # Tell daemon to spawn node process -├── stop # Tell daemon to stop node process -├── list # Show all nodes and their status -└── status # Detailed status/metrics for one node +ant [global flags] +├── file +│ ├── upload [--public] [--merkle|--no-merkle] +│ └── download [ADDRESS] [--datamap PATH] [-o PATH] +├── chunk +│ ├── put [FILE] +│ └── get
[-o PATH] +├── wallet +│ ├── address +│ └── balance +└── node + ├── daemon start + ├── daemon stop + ├── daemon status + ├── daemon info + ├── daemon run # Hidden; runs daemon in foreground + ├── add [options] + ├── start [--service-name NAME] + ├── stop [--service-name NAME] + ├── status + └── reset [--force] ``` The `--json` global flag causes all commands to output structured JSON instead of human-readable text. This is essential for AI agents and scripts that consume the CLI directly rather than using the REST API. +## Data Flow: Upload and Download + +### File Upload + +``` +┌─────────┐ read 8KB ┌──────────────────┐ encrypted ┌──────────┐ +│ Disk │──────────────▶│ Self-Encryption │──────chunks───▶│ Client │ +│ (file) │ streaming │ (convergent enc) │ │ │ +└─────────┘ └──────────────────┘ └──────────┘ + │ │ + ▼ │ for each chunk: + DataMap │ 1. get quotes + (chunk addresses │ 2. pay (EVM tx) + + encryption keys) │ 3. store on peer + ▼ + ┌────────────────┐ + │ P2P Network │ + │ (XOR-routed) │ + └────────────────┘ +``` + +- Files are streamed in 8KB reads through `self_encryption::stream_encrypt`. +- Each encrypted chunk is content-addressed (XOR hash of its bytes). +- Payment is per-chunk or via a merkle batch (single EVM transaction for many chunks). +- The `DataMap` is the key to reconstruct the file. Store it publicly (on-network) or keep it private (local file). + +### File Download + +``` +┌───────────┐ chunk addresses ┌──────────┐ requests ┌────────────────┐ +│ DataMap │──────────────────▶│ Client │─────────────▶│ P2P Network │ +└───────────┘ │ │◀─────────────│ (XOR-routed) │ + └──────────┘ encrypted └────────────────┘ + │ chunks + ▼ + ┌──────────────────┐ + │ Self-Decryption │ + │ (streaming) │ + └──────────────────┘ + │ + ▼ + ┌─────────┐ + │ Disk │ + │ (file) │ + └─────────┘ +``` + +- Downloads are also streaming: chunks are fetched concurrently and decrypted in batches. +- The file is written incrementally via a temp file, then atomically renamed. + ## Node Registry A JSON file persisted at a platform-appropriate location (e.g., @@ -142,9 +218,10 @@ their configuration. "nodes": { "1": { "id": 1, + "service_name": "node1", "rewards_address": "0xabc123...", - "data_dir": "/home/user/.local/share/ant/nodes/1", - "log_dir": "/home/user/.local/share/ant/nodes/1/logs", + "data_dir": "/home/user/.local/share/ant/nodes/node-1", + "log_dir": "/home/user/.local/share/ant/nodes/node-1/logs", "node_port": null, "metrics_port": null, "network_id": 1, @@ -179,17 +256,17 @@ AI agents and other HTTP clients need to find the daemon. Two mechanisms: ``` GET /api/v1/status Daemon health, uptime, node count summary -GET /api/v1/nodes List all nodes with status -GET /api/v1/nodes/:id Node detail including runtime metrics -POST /api/v1/nodes Add/register a new node (same as CLI `add`) +GET /api/v1/nodes/status Node status summary (all nodes) +POST /api/v1/nodes Add/register nodes (same as CLI `add`) DELETE /api/v1/nodes/:id Remove a node from the registry POST /api/v1/nodes/:id/start Start a node POST /api/v1/nodes/:id/stop Stop a node POST /api/v1/nodes/start-all Start all registered nodes POST /api/v1/nodes/stop-all Stop all running nodes +POST /api/v1/reset Reset all node state (fails if nodes running) GET /api/v1/events SSE stream of real-time node events GET /api/v1/openapi.json OpenAPI spec for API self-discovery -GET / Web status console (static HTML) +GET /console Web status console (static HTML) ``` ### Idempotency @@ -233,8 +310,7 @@ All error responses use a consistent envelope: The daemon serves an OpenAPI 3.1 spec at `/api/v1/openapi.json`. This allows AI agents to self-discover available endpoints, parameter shapes, and response schemas without external -documentation. The spec is generated from the axum route definitions using `utoipa` or a similar -library. +documentation. The spec is generated from the axum route definitions using `utoipa`. ## Process Management @@ -310,4 +386,4 @@ event: node_crashed data: {"node_id": 2, "exit_code": 1} ``` -Clients that don't need real-time updates can poll `GET /api/v1/nodes` instead. +Clients that don't need real-time updates can poll `GET /api/v1/nodes/status` instead.