Skip to content

feat: add S3-compatible storage interface (Layer 1)#19

Open
mudigal wants to merge 3 commits intofeat/file-systemfrom
feat/s3-storage-interface
Open

feat: add S3-compatible storage interface (Layer 1)#19
mudigal wants to merge 3 commits intofeat/file-systemfrom
feat/s3-storage-interface

Conversation

@mudigal
Copy link
Collaborator

@mudigal mudigal commented Feb 26, 2026

Summary

  • Add S3-compatible storage interface that provides familiar S3 API semantics on top of Layer 0 blob storage
  • S3 pallet internally creates Layer 0 buckets, encapsulating the storage layer from clients
  • Clients only interact with the S3 layer while Layer 0 provides underlying storage guarantees (checkpoints, challenges, slashing)

Components Added

Component Description
s3-primitives Core types (ObjectMetadata, S3BucketInfo) and validation functions
pallet-s3-registry On-chain S3 bucket/object metadata with automatic Layer 0 bucket creation
s3-client High-level SDK using subxt 0.44 for chain operations

Key Features

  • Bucket operations: create, delete, head, list
  • Object operations: put, get, delete, copy, list with prefix/delimiter support
  • S3 naming rules validation (3-63 chars, lowercase alphanumeric + hyphens)
  • Content-addressed storage via CID (blake2-256)
  • User-defined metadata support

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    S3 Client SDK                            │
│    (put_object, get_object, list_objects, etc.)             │
└─────────────────────────────────────────────────────────────┘
        │                                      │
        ▼                                      ▼
┌───────────────────────┐    ┌─────────────────────────────────┐
│   pallet-s3-registry  │    │   Provider Node (Layer 0)       │
│   (On-chain metadata) │    │   (Unchanged - blob storage)    │
└───────────────────────┘    └─────────────────────────────────┘

Test plan

  • cargo test -p s3-primitives - 3 tests pass
  • cargo test -p pallet-s3-registry - 10 tests pass
  • cargo test -p s3-client - 1 test passes
  • Integration test with running infrastructure (just s3-example)

🤖 Generated with Claude Code

Implement S3-compatible storage interface that provides familiar S3 API
semantics on top of Layer 0 blob storage. The S3 pallet internally creates
Layer 0 buckets, encapsulating the storage layer from clients.

Components:
- s3-primitives: Core types (ObjectMetadata, S3BucketInfo) and validation
- pallet-s3-registry: On-chain S3 bucket/object metadata with automatic
  Layer 0 bucket creation via create_bucket_internal()
- s3-client: High-level SDK using subxt 0.44 for chain operations and
  storage-client for provider blob operations

Key features:
- Bucket operations: create, delete, head, list
- Object operations: put, get, delete, copy, list with prefix/delimiter
- S3 naming rules validation (3-63 chars, lowercase alphanumeric + hyphens)
- Content-addressed storage via CID (blake2-256)
- User-defined metadata support

The architecture ensures clients only interact with the S3 layer while
Layer 0 provides the underlying storage guarantees (checkpoints, challenges,
slashing).
@mudigal mudigal changed the base branch from dev to feat/file-system February 26, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant