Skip to content

feat(storage): add named S3-compatible backends via s3+<name>:// URIs#1

Merged
papaharry merged 1 commit into
mainfrom
feature/s3-named-backends
Jun 10, 2026
Merged

feat(storage): add named S3-compatible backends via s3+<name>:// URIs#1
papaharry merged 1 commit into
mainfrom
feature/s3-named-backends

Conversation

@papaharry

Copy link
Copy Markdown
Collaborator

Summary

  • Adds `storage.s3.named.` config map so a single Quickwit cluster can address multiple S3-compatible backends (separate creds, region, endpoint, checksum settings, flavor flags per entry).
  • URIs of the form `s3+://bucket/path` route to the named backend; plain `s3://` continues to use the primary endpoint.
  • Preserves the `s3+` qualifier through `Uri::parse_str` normalization so the storage resolver can recover the backend name on deserialization. Without this, named URIs were silently rewritten to plain `s3://` and resolved to the primary endpoint (silently wrong routing).

Why

The PoC needed A/B comparisons across Civo S3, SeaweedFS S3 (in-cluster), and OVH Object Storage on the same Quickwit cluster. Upstream Quickwit supports a single S3-compatible endpoint per cluster. This change makes multi-backend addressable per-index without forking the storage layer.

Changes

  • `quickwit-config/src/storage_config.rs`: `NamedS3StorageConfig` struct + `storage.s3.named` BTreeMap, projection back to `S3StorageConfig` via `as_s3_config()`, redact + Debug + checksum_algorithm passthrough.
  • `quickwit-common/src/uri.rs`: preserve `s3+` scheme through parse_str (so the named qualifier survives serde roundtrip via the metastore).
  • `quickwit-storage/src/object_storage/s3_compatible_storage_resolver.rs`: `parse_named_key`, per-name cached S3Client, dispatch in `resolve()`.
  • `docs/configuration/storage-config.md`: list `s3+://` in the URI protocols + Named S3 backends section with YAML example.

Tests

  • `uri::tests::test_uri_s3_named_preserved` — parse + serde roundtrip preserves the qualifier
  • `storage_config::tests::test_storage_s3_named_backends_serde` — YAML deser, `as_s3_config` projection
  • `storage_config::tests::test_storage_s3_named_backends_redact`
  • `s3_compatible_storage_resolver::tests::test_parse_named_key`

Test plan

  • `cargo test -p quickwit-common --lib uri::`
  • `cargo test -p quickwit-config --lib storage_config::`
  • `cargo test -p quickwit-storage --lib s3_compatible_storage_resolver::`
  • End-to-end smoke against a running cluster: created indexes with `s3+seaweedfs://`, `s3+ovh-morocco://`, `s3+civo-fra1://` URIs; verified ingest/search routes to the correct backend; verified URIs are stored intact in the metastore.

🤖 Generated with Claude Code

- storage.s3.named.<name> config map for arbitrary S3-compatible
  endpoints (separate creds, region, endpoint, checksum, flags per entry).
- URIs of the form s3+<name>://bucket/path route to the named backend;
  plain s3:// continues to use the primary endpoint.
- Preserve the s3+<name> qualifier through Uri::parse_str normalization
  so the resolver can recover the backend name on deserialization
  (previously the qualifier was stripped and named URIs silently
  resolved to the primary endpoint).
- Tests: URI parse/serde roundtrip, NamedS3StorageConfig serde + redact +
  as_s3_config projection, parse_named_key.
- Docs: list s3+<name>:// in the URI protocols and add a Named S3
  backends section with a YAML example.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@papaharry papaharry merged commit 78dd70e into main Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant